Indexing 100M vectors in 20 minutes on PostgreSQL with 12GB RAM

4 comments

• nwellinghoff4 days ago
  Too bad aws does not support any of these other vector extensions in managed rds.
• esafak2 hours ago
  How does it compare with paradedb and lancedb?
• ayende2 hours ago
  That suffer from a serious issue<p>You must have the data upfront, you cannot build this in an incremental fashion<p>There is also bo mention on how this would handle updates, and from the description, even if updates are possible, this will degrade over time, requiring new indexing batch
• duckbot30004 days ago
  Kinda makes you wonder why you need cloud for anything besides remote encrypted backups if you can run all that on 12GB
  • riku_iki3 days ago
    what about failover story if server dies? PG failover setup is complicated, and cloud infra handles this for you.
    • tjwebbnorfolk33 minutes ago
      What are you willing to pay for cloud-native failover?<p>Not every use case requires 100% uptime
    • logifail1 hour ago
      (Genuine question) What&#x27;s your current plan for when your cloud provider goes offline? Do you have a failover story, or it a case of &quot;wait for them to come back online&quot;?
    • benjiro14 minutes ago
      <a href="https:&#x2F;&#x2F;github.com&#x2F;multigres&#x2F;multigres" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;multigres&#x2F;multigres</a> ... when its complete. From the guy that made Vitess for Mysql.<p>And yes, i agree, the PG failover setup (and especially dealing with a failure afterwards, to restore the ex-master is beyond infuriating).<p>But its not pay 10x the amount, while eating easily 10x performance infuriating :)
    • positron262 hours ago
      Do we mean managed or PG on K8s like CNPG? In all cases, I use the infra to simplify things like having disk redundancy and failover nodes, not because 12GB is interesting.
      • riku_iki2 hours ago
        Primary managed PG, since you still need setup&#x2F;maintenance&#x2F;monitoring on your K8S own solution.
  • setr4 days ago
    Because getting any hardware out of infra-team on premise is utterly miserable, across the board.
    • lelanthran2 hours ago
      That&#x27;s not the only alternative.<p>Rent your VPS and add in extra volumes for like $10 per 100GB.
      • Imustaskforhelp21 minutes ago
        Funny thing but netcup has $10 per 1 TB<p>Netcup is under-rated but there are also other providers too at lowendbox&#x2F;lowendtalk and I am interested to try out hetzner too sometime.
        • benjiro9 minutes ago
          And if you want to go even cheaper, check out Hetzner their EX63 (go to custom) &gt; 4x 7.68TB drives for like 140 Euro.<p>Not counting the fact that Netcup is raided (also Netcup is limited to 8TB on a VPS).<p>That is like 4.7 Euro &#x2F;TB. That is like 4$&#x2F;TB. 6 Euro &#x2F; TB in a raid 5 setup.<p>I do not understand why they are not using this new pricing model on their older servers. There the best you can get is like 10 Euro &#x2F;TB (for the single 15TB U.2).