Postgres extension complements pgvector for performance and scale

(github.com)

93 points by flyaway1235 days ago

5 comments

ricw4 hours ago
I’ve been using this since early this year and it’s been great. It was what convinced me to just stick to Postgres rather than using a dedicated vector db.Only working with 100m or so vectors, but for that it does the job.
- pqdbr4 hours ago
 Are you using a dedicated pg instance for vector or you keep all your data in a single pg instance (vector and non-vector)?
 - ComputerGuru4 hours ago
 The biggest selling point to using Postgres over qdrant or whatever is that you can put all the data in the same db and use joins and ctes, foreign keys and other constraints, lower latency, get rid of effectively n+1 cases, and ensure data integrity.
 - dalberto3 hours ago
 I generally agree that one database instance is ideal, but there are other reasons why Postgres everywhere is advantageous, even across multiple instances:- Expertise: it's just SQL for the most part - Ecosystem: same ORM, same connection pooler - Portability: all major clouds have managed PostgresI'd gladly take multiple Postgres instances even if I lose cross-database joins.
 - throwaway77833 hours ago
 Yep. If performance becomes a concern, but we still want to exploit joins etc, it's easy to set up replicas and "shard" read only use cases across replicas.
 - ricw1 hour ago
 All in one of course. That’s the biggest advantage. And why postgres is great - it covers virtually all standard use cases.
- esafak4 hours ago
 What kind of performance do you observe with what setup?
 - ricw1 hour ago
 Depends on the query and I don’t have exact numbers of the top of my head, but we’re talking low 100ms range for something pgvector itself wasn’t able to handle in a reasonable amount of time.
aunty_helen3 hours ago
Related discussion for pgvector perf: <a href="https://news.ycombinator.com/item?id=45798479">https://news.ycombinator.com/item?id=45798479</a>
- tacoooooooo3 hours ago
 the main issue with pgvectorscale is that it's not available in RDS :(
 - mrinterweb32 minutes ago
 I'm considering hosting a separate pg db just to be able to access certain extensions. I am interested in this extension as well as <a href="https://wiki.postgresql.org/wiki/Incremental_View_Maintenance" rel="nofollow">https://wiki.postgresql.org/wiki/Incremental_View_Maintenanc...</a> (also not available on RDS). Then use logical replication for specific data source tables (guess it would need to be DMS).
 - omg28642 hours ago
 Yes, RDS seems to really hold PG back on AWS, with all the interesting pg extensions getting released now (pg_lake). It is a share I can't move to other PG vendors because it is a pain in the ass to get all privacy, legal docs in order.
 - calderwoodra48 minutes ago
 Yes, the InfoSec advantages of using RDS are very real, especially in B2B Enterprise SaaS.
jascha_eng1 hour ago
Combined with our other search extension for full text search these two extensions make postgres a really capable hybrid search engine: <a href="https://github.com/timescale/pg_textsearch" rel="nofollow">https://github.com/timescale/pg_textsearch</a>
- ldng21 minutes ago
 I'm not how you'd combine the two; care to give us a quick outline ?
isoprophlex3 hours ago
The linked blogpost is an interesting read, too, comparing well-tuned pgvector to pinecone:<a href="https://www.tigerdata.com/blog/pgvector-vs-pinecone" rel="nofollow">https://www.tigerdata.com/blog/pgvector-vs-pinecone</a>
mmmeff3 hours ago
This is still unsupported in RDS, right?
- jascha_eng1 hour ago
 We have a lot of happy customers that moved from rds to tiger cloud if you think pgvectorscale is interesting to you and you don't want to self host pg.But yes big cloud providers move slow in adopting extensions.
- tacoooooooo3 hours ago
 correct afaik :(<a href="https://github.com/timescale/pgvectorscale/issues/113" rel="nofollow">https://github.com/timescale/pgvectorscale/issues/113</a>