Do people have good experiences with LMDB, in terms of reliability? I've never used it in production, but I've read through the code and design documents for a database implementation class.<p>I remember some strange code (such as pushing return values 4k above the stack, with a comment like "this works as long as the caller doesn't use more than 4k of stack space before accessing the return value"), and the author also shared some unconventional opinions about undefined behavior (like "Compilers are deterministic, if I know what platform I'm compiling to then no behavior is undefined. And if compiler authors disagree, they are morons.")<p>But presumably it's thoroughly tested, so those aren't problems in practice? Would be really interested to hear from people who've actually used it. I've mainly stuck to SQLite instead.
Not amazing. In certain workloads I ran, once the db reached several hundred gb, writes would hang for longer and longer periods of time, eventually hours, while the db grew drastically in the background. <a href="https://news.ycombinator.com/item?id=30023623">https://news.ycombinator.com/item?id=30023623</a> seems to be the same issue, and it was serious enough that Shopify decided not to use lmdb.<p>And yes, I ensured there were no outstanding long lived readers, verified with mdb_stat -r. My workload used one transaction per read/write anyway (never needed larger atomicity). Once the db got into the bad state, running my program on it would almost immediately run into the issue again, so I really think the db is in a bad state such that most writes would cause it to hang, not related to how I do transactions. This workload would pretty consistently hit the issue once the db got to several hundred gb.<p>Issue #10236 on the OpenLDAP bug tracker might be the root cause, who knows. It's been marked CONFIRMED for years without a fix, while other similar issues are created.<p>This is extremely annoying. It seems workload dependent (other workloads I've run create absolutely massive lmdb dbs without this issue) and once it happens your only recourse is to make a new db and copy the contents over (thankfully reads still work fine on these borked dbs).<p>Other than that, though, it's great. Never in any case had actual data corruption, and reads and writes are extremely fast (until this issue happens)<p>Edit: fun fact, since shopify may have created Bolt in response to this bug, and then Bolt was the root cause of the 73-hour Roblox downtime in 2021, this bug may indirectly have caused one of the worst outages ever!
That it keeps an infinite cache of malloc page allocations is annoying (the issue you referenced). I just removed that (after complaining on the mailing list about it). The performance advantage is probably negligible in many cases (since malloc implementations often already cache), while causing confusing memory usage behavior.<p>Idk, if it was your issue, but for long running write transactions it doesn't spill to disk. So you have all the changes being written to disk at the end of the transaction. One would think enabling write mapping fixes this, but it needs to mark all the pages as clean before commit, so same effect there.
I fixed this for 0.9 here <a href="https://github.com/uroni/hs5/tree/main/external/lmdb" rel="nofollow">https://github.com/uroni/hs5/tree/main/external/lmdb</a> . Will have to investigate if it is improved with 1.0, or if I need to redo the changes.<p>Edit: Just noticed that the issue is about free list in the file. Never had a problem with that, but I also had to replace that MIDL structure with something more scalable for the spilling.
I've used LMDB in production for multi-terabyte databases, and we encountered the long-write time but found a solution.<p>The important idea is that LMDB offloads cache management almost completely to the OS. You have to become intimately familiar with the way that the page cache works and how to configure it.
I think lmdb is mostly unusable, for many use cases. I switched to libmdbx, which fixes all the issues [2] I (and most sibling comments) ran into with lmdb.<p>[1] <a href="https://github.com/Mithril-mine/libmdbx#improvements-beyond-lmdb" rel="nofollow">https://github.com/Mithril-mine/libmdbx#improvements-beyond-...</a>
I can't go into specifics, but I use LMDB for the commandline application I maintain for my employer. I also extended it into a web service for internal use. As long as you stick to the safe LMDB options, which are the default options, it's reliable. The documentation clearly outlines what safety guarantees you lose when you enable/disable certain options: <a href="http://www.lmdb.tech/doc/group__mdb.html#ga32a193c6bf4d7d5c5d579e71f22e9340" rel="nofollow">http://www.lmdb.tech/doc/group__mdb.html#ga32a193c6bf4d7d5c5...</a><p>I had a situation where the web service's writes were slowing down to an unbearable crawl because the number of entries in the database were reaching tens of billions of entries. Thankfully, the users never experienced the slowness. The website stayed nice and fast, even though the background updates were extraordinarily slow. The issue was fixed by sharding the databases.
I use it as a session store for a computer music system. It has worked well for me as a way to read mutable (by any client) parameters during synthesis, clients will often read dozens of parameters during a block of computation (a relatively short window of time in the low milliseconds typically) without adding any noticeable overhead to the render time for each block.<p>Edit: I also tried using it for larger blobs of data (like audio) but ended up only storing a reference to shared memory for larger blocks, anything larger than IIRC 4k that can't be stored in a single node kills performance, but for small values it seems pretty great.
It has been used successfully as the backend for OpenLDAP and Monero, at least.
Be cautious if you're using large databases on iOS. At least until fairly recently, iOS doesn't page dirty mmaped pages back to disk and after enough churn the app will OOM.
I believe at least one of the two official Minecraft implementations use it for their map/save format.
We evaluated it but chose RocksDB instead
It is a small amount of code so easy to integrate into an application.<p>It is really reliable except write performance in my experience.<p>Author of it writes very spicy stuff and sounds pretty rude.<p>I would recommend doing a prototype with real data scale and testing if it meets your requirements. The write performance can be really atrocious and It doesn't have a high performance potential because it is based on memmap.