What a bizarre conclusion to draw! It's like saying that cars are the best means of transportation because you can travel to the Grand Canyon in them and the Grand Canyon is the best landscape in the world, and yes you could use other means to get there, but cars are what everybody's using.<p>If the real goal of TFA was to praise C's ability to reinterpret a chunk of memory (possibly mapped to a file) as another datatype, it would have been more effective to do so using C functions and not OS-specific system calls. For example:<p><pre><code> FILE *f = fopen(...);
uint32_t *numbers;
fread(numbers, ..., f);
access numbers[...]
frwite(numbers, ..., f);
fclose(f);</code></pre>
This is way more cumbersome than mmap if you need to out-of-core process the file in non-sequential patterns. Way way more cumbersome, since you need to deal with intermediate staging buffers, and reuse them if you actually want to be fast. mmap, on the other hand, is absolutely trivial to use, like any regular buffer pointer. And at least on windows, the mmap counterpart can be faster when processing the file with multiple threads, compared to fread.<p>But I agree that it's a bizarre article since mmap is not a C standard, and relies on platform-dependend Operating System APIs.
I can’t entirely tell what the article’s point is. It seems to be trying to say that many languages can mmap <i>bytes</i>, but:<p>> (as far as I'm aware) C is the only language that lets you specify a binary format and just use it.<p>I assume they mean:<p><pre><code> struct foo { fields; };
foo *data = mmap(…);
</code></pre>
And yes, C is one of relatively few languages that let you do this without complaint, <i>because it’s a terrible idea</i>. And C doesn’t even let you specify a binary format — it lets you write a struct that will correspond to a binary format in accordance with the C ABI on your particular system.<p>If you want to access a file containing a bunch of records using mmap, and you want a well defined format and good performance, then use something actually intended for the purpose. Cap’n Proto and FlatBuffers are fast but often produce rather large output; protobuf and its ilk are more space efficient and very widely supported; Parquet and Feather can have excellent performance and space efficiency if you use them for their intended purposes. And <i>everything</i> needs to deal with the fact that, if you carelessly access mmapped data that is modified while you read it in any C-like language, you get UB.
> correspond to a binary format in accordance with the C ABI on your particular system.<p>We're so deep in this hole that people are fixing this on a CPU with silicon.<p>The Graviton team made a little-endian version of ARM just to allow lazy code like this to migrate away from Intel chips without having to rewrite struct unpacking (& also IBM with the ppc64le).<p>Early in my career, I spent a lot of my time reading Java bytecode into little endian to match all the bytecode interpreter enums I had & completely hating how 0xCAFEBABE would literally say BE BA FE CA (jokingly referred as "be bull shit") in a (gdb) x views.
Had the same thought. Also confused at the backhanded compliment that pickle got:<p>> Just look at Python's pickle: it's a completely insecure serialization format. Loading a file can cause code execution even if you just wanted some numbers... but still very widely used because it fits the mix-code-and-data model of python.<p>Like, are they saying it's bad? Are they saying it's good? I don't even get it. While I was reading the post, I was thinking about pickle the whole time (and how terrible <i>that</i> idea is, too).
Why is it such a terrible idea?<p>No need to add complexity, dependancies and reduced performance by using these libraries.
Lots of reasons:<p>The code is not portable between architectures.<p>You can’t actually define your data structure. You can pretend with your compiler’s version of “pack” with regrettable results.<p>You probably have multiple kinds of undefined behavior.<p>Dealing with compatibility between versions of your software is awkward at best.<p>You might not even get amazing performance. mmap is not a panacea. Page faults and TLB flushing are not free.<p>You can’t use any sort of advanced data types — you get exactly what C gives you.<p>Forget about enforcing any sort of invariant at the language level.
I've written a lot of code using that method, and never had any portability issues. You use types with number of bits in them.<p>Hell, I've slung C structs across the network between 3 CPU architectures. And I didn't even use htons!<p>Maybe it's not portable to some ancient architecture, but none that I have experienced.<p>If there is undefined behavior, it's certainly never been a problem either.<p>And I've seen a lot of talk about TLB shootdown, so I tried to reproduce those problems but even with over 32 threads, mmap was still faster than fread into memory in the tests I ran.<p>Look, obviously there are use cases for libraries like that, but a lot of the time you just need something simple, and writing some structs to disk can go a long way.
No defined binary encoding, no guarantee about concurrent modifications, performance trade-offs (mmap is NOT always faster than sequential reads!) and more.
Because a struct might not serialize the same way from a CPU architecture to another.<p>The size of int can be different, the padding can be different, etc.
mmap is not a C feature, but POSIX. There are C platforms that don't provide mmap, and on those that do you can use mmap from other languages (there's mmap module in the Python's standard library, for example).
I think this is sort of missing the point, though. Yes, mmap() is in POSIX[1] in the sense of "where is it specified".<p>But mmap() <i>was implemented in C</i> because C is the natural language for exposing Unix system calls and mmap() is a syscall provided by the OS. And this is true up and down the stack. Best language for integrating with low level kernel networking (sockopts, routing, etc...)? C. Best language for async I/O primitives? C. Best language for SIMD integration? C. And it goes on and on.<p>Obviously you can do this stuff (including mmap()) in all sorts of runtimes. But it always appears first in C and gets ported elsewhere. Because no matter how much you think your language is better, if you have to go into the kernel to plumb out hooks for your new feature, you're going to integrated and test it using a C rig before you get the other ports.<p>[1] Given that the pedantry bottle was opened already, it's worth pointing out that you'd have gotten more points by noting that it appeared in 4.2BSD.
If we're going to be pedantic, mmap is a syscall. It happens that the C version is standardized by POSIX.<p>The underlying syscall doesn't use the C ABI, you need to wrap it to use it from C in the same way you need to wrap it to use it from any language, which is exactly what glibc and friends do.<p>Moral of the story is mmap belongs to the platform, not the language.
Why does Ada have the best file API?<p><a href="https://github.com/AdaCore/florist/blob/master/libsrc/posix-memory_mapping.ads" rel="nofollow">https://github.com/AdaCore/florist/blob/master/libsrc/posix-...</a>
Using mmap means that you need to be able to handle memory access exceptions when a disk read or write fails. Examples of disk access that fails includes reading from a file on a Wifi network drive, a USB device with a cable that suddenly loses its connection when the cable is jiggled, or even a removable USB drive where all disk reads fail after it sees one bad sector. If you're not prepared to handle a memory access exception when you access the mapped file, don't use mmap.
You can even mmap a socket on some systems (iOS and macOS via GCD). But doing that is super fragile. Socket errors are swallowed.<p>My interpretation always was the mmap should only be used for immutable and local files. You may still run into issues with those type of files but it’s very unlikely.
C doesn't have exceptions, do you mean signals? If not, I don't see how that is that any different from having to handle I/O errors from write() and/or open() calls.
Ah, reminds me of 'Are You Sure You Want to Use MMAP in Your Database Management System? (2022)' <a href="https://db.cs.cmu.edu/mmap-cidr2022/" rel="nofollow">https://db.cs.cmu.edu/mmap-cidr2022/</a>
I think C# standard library is better. You can do same unsafe code as in C, SafeBuffer.AcquirePointer method then directly access the memory. Or you can do safer and slightly slower by calling Read or Write methods of MemoryMappedViewAccessor.<p>All these methods are in the standard library, i.e. they work on all platforms. The C code is specific to POSIX; Windows supports memory mapped files too but the APIs are quite different.
I think you don’t need to be unsafe, they have normal API for it.<p><a href="https://learn.microsoft.com/en-us/dotnet/standard/io/memory-mapped-files" rel="nofollow">https://learn.microsoft.com/en-us/dotnet/standard/io/memory-...</a>
Indeed, but these normal APIs have runtime costs for bounds checking. For some use cases, unsafe can be better. For instance, last time I used a memory-mapped file was for a large immutable Bloom filter. I knew the file should be exactly 4GB, validated that in the constructor, then when testing 12 bits from random location of the mapped file on each query, I opted for unsafe codes.
Aside from what <a href="https://news.ycombinator.com/item?id=47210893">https://news.ycombinator.com/item?id=47210893</a> said, mmap() is a low-level design that makes it easier to work with files that don't fit in memory and fundamentally represent a single homogeneous array of some structure. But it turns out that files commonly <i>do</i> fit in memory (nowadays you commonly have on the order of ~100x as much disk as memory, but millions of files); and you very often <i>want</i> to read them in order, because that's the easiest way to make sense of them (and tape is <i>not at all</i> the only storage medium historically that had a much easier time with linear access than random access); and you <i>need</i> to parse them because they <i>don't</i> represent any such array.<p>When I was first taught C formally, they definitely walked us through all the standard FILE* manipulators and didn't mention mmap() at all. And when I first heard about mmap() I couldn't imagine personally having a reason to use it.
> But it turns out that files commonly do fit in memory<p>The difference between slurping a file into malloc'd memory and just mmap'ing it is that the latter doesn't use up anonymous memory. Under memory pressure, the mmap'd file can just be evicted and transparently reloaded later, whereas if it was copied into anonymous memory it either needs to be copied out to swap or, if there's not enough swap (e.g. if swap is disabled), the OOM killer will be invoked to shoot down some (often innocent) process.<p>If you need an entire file loaded into your address space, and you don't have to worry about the file being modified (e.g. have to deal with SIGBUS if the file is truncated), then mmap'ing the file is being a good citizen in terms of wisely using system resources. On a system like Linux that aggressively buffers file data, there likely won't be a performance difference <i>if</i> your memory usage assumptions are correct, though you can use madvise & friends to hint to the kernel. If your assumptions are wrong, then you get graceful performance degradation (back pressure, effectively) rather than breaking things.<p>Are you tired of bloated software slowing your systems to a crawl because most developers and application processes think they're special snowflakes that will have a machine all to themselves? Be part of the solution, not part of the problem.
mmap is also relatively slow (compared to modern solutions, io_uring and friends), and immensely painful for error handling.<p>It's simple, I'll give it that.
I guess the author didn't use that many other programming languages or OSes. You can do the same even in garbage collected languages like Java and C# and on Windows too.<p><a href="https://docs.oracle.com/javase/8/docs/api/java/nio/MappedByteBuffer.html" rel="nofollow">https://docs.oracle.com/javase/8/docs/api/java/nio/MappedByt...</a><p><a href="https://learn.microsoft.com/en-us/dotnet/api/system.io.memorymappedfiles.memorymappedfile?view=net-10.0" rel="nofollow">https://learn.microsoft.com/en-us/dotnet/api/system.io.memor...</a><p><a href="https://learn.microsoft.com/en-us/windows/win32/memory/creating-a-file-mapping-object" rel="nofollow">https://learn.microsoft.com/en-us/windows/win32/memory/creat...</a><p>Memory mapping is very common.
Well...<p>I'm not sure what the author really wants to say. mmap is available in many languages (e.g. Python) on Linux (and many other *nix I suppose). C provides you with raw memory access, so using mmap is sort-of-convenient for this use case.<p>But if you use Python then, yes, you'll need a bytearray, because Python doesn't give you raw access to such memory - and I'm not sure you'd want to mmap a PyObject anyway?<p>Then, writing and reading this kind of raw memory can be kind of dangerous and non-portable - I'm not really sure that the pickle analogy even makes sense. I very much suppose (I've never tried) that if you mmap-read malicious data in C, a vulnerability would be _quite_ easy to exploit.
Creating memory mapped files is a very common OS feature since 90s. Many high level languages have it as OS agnostic POSIX or not.
> very common OS feature since 90s<p>And if you want to go farther back, even if it wasn't called "mmap" or a specific function you had to invoke -- there were operating systems that used a "single-level store" (notably MULTICS and IBM's AS/400..err OS/400... err i5 OS... err today IBM i [seriously, IBM, pick a name and stick with it]) where the interface to disk storage on the platform is that the entire disk storage/filesystem is always mapped into the same address space as the rest of your process's memory. Memory-mapped files were basically the only interface there was, and the operating system "magically" persisted certain areas of your memory to permanent storage.
Actually in Python you could recast (zerocopy) bytearray as other primitive C type or even any other structure using ctypes module.
It may have a tidy mmap api, but Smalltalk has a much better file api through its Streams hierarchy IMHO. You can create a stream on a diskfile, you can create a stream on a byteArray, you can create a stream on standard Unix streams, you can create a stream on anything where "next" makes sense.
> Why does C have the best file API<p>> Look inside<p>> Platform APIs<p>Ok.<p>I agree platform APIs are better than most generic language APIs at least. I disagree on mmap being the "best".
I think OP and I have very divergent opinions on what makes a file API "best". This may have been the best 30 years ago. The world has moved on.
The article only touches on `open` and `close` and doesn't deal with any of the realities of file access. Not a particularly compelling write-up.
> However, in other most languages, you have to read() in tiny chunks, parse, process, serialize and finally write() back to the disk. This works, but is verbose and needlessly limited<p>C has those too and am glad that they do. This is what allows one to do other things while the buffer gets filled, without the need for multithreading.<p>Yes easier standardized portable async interfaces would have been nice, not sure how well supported they are.
It has the best API for the author, that's for sure. One size does not fit all: believe it or not, different files have different uses. One does not mmap a pipe or /dev/urandom.
"best file API" and the man page for the O_ flags disagree.
A file API is not the same thing as a <i>filesystem</i> API. The holy grail is still a universal but high(-enough)-level filesystem API.
mmap() is useful for some narrow use-cases, I think, but error-handling is a <i>huge</i> pain. I don't want to have to deal with SIGBUS in my application.<p>I agree that the <i>model</i> of mmap() is amazing, though: being able to treat a file as just a range of bytes in memory, while letting the OS handle the fussy details, is incredibly useful. (It's just that the OS doesn't handle <i>all</i> of the fussy details, and that's a pain.)
C's API does not include mmap, nor does it contain any API to deal with file paths, nor does it contain any support for opening up a file picker. This paired with C's bad string support results in one of it being one of the worst file APIs.<p>Also using mmap is not as simple as the article lays out. For example what happens when another process modifies the file and now your processes' mapped memory consists of parts of 2 different versions of the file at the same time. You also need to build a way to know how to grow the mapping if you run out room. You also want to be able to handle failures to read or write. This means you pretty much will need to reimplement a fread and fwrite going back to the approach the author didn't like: "This works, but is verbose and needlessly limited to sequential access." So it turns out "It ends up being just a nicer way to call read() and write()" is only true if you ignore the edge cases.
mmap is nice. But, I find sqlite is a better filesystem API [1]. If you are going to use mmap why not take it further and use LMDB? Both have bindings for most languages.<p>[1] - <a href="https://sqlite.org/fasterthanfs.html" rel="nofollow">https://sqlite.org/fasterthanfs.html</a>
technically yes, because there's a failure path for every single failure that an OS knows about. And most others aren't so resilient. However, mmap bypasses a lot of that....
How do you handle read/write errors with mmap?
mmap on file io errors would manifest in Signals (For example SIGBUS or SIGSEGV).<p>So if you wanted to handle file read/write errors you would need to implement signal handlers.<p><a href="https://stackoverflow.com/questions/6791415/how-do-memory-mapped-files-handle-i-o-errors" rel="nofollow">https://stackoverflow.com/questions/6791415/how-do-memory-ma...</a>
In my experience, having worked with a large system that used almost exclusively mmap for I/O, you don’t. The process segfaults and is restarted. In practice it almost never happened.
Here's a negative signal I'm seeing often:<p>When a developer that usually consumes the language starts critiquing the language.<p>I could go on as to why it's a bad signal, psychologically, but let's just say that empirically it usually doesn't come from a good place, it's more like a developer raising the stakes of their application failing and blaming the language.<p>Sure one out of a thousand devs might be the next Linus Torvalds and develop the next Rust. But the odds aren't great.
mmap is not a language feature. it is also full of its own pitfalls that you need to be aware of. recommended reading: <a href="https://db.cs.cmu.edu/mmap-cidr2022/" rel="nofollow">https://db.cs.cmu.edu/mmap-cidr2022/</a>
This is like: I discovered the wheel and want to let you know!
<i>It still works if the file doesn't fit in RAM</i><p>No it doesn't. If you have a file that's 2^36 bytes and your address space is only 2^32, it won't work.<p>On a related digression, I've seen so many cases of programs that could've handled <i>infinitely</i> long input in constant space instead implemented as some form of "read the whole input into memory", which unnecessarily puts a limit on the input length.
Address space size and RAM are two different things.
You <i>can</i> mmap with offset, for that case. Just FYI in anyone thought it was a hard limit.
All memory map APIs support moveable “windows” or views into files that are much larger than either physical memory or the virtual address space.<p>I’ve seen otherwise competent developers use compile time flags to bypass memmap on 32-bit systems even though this always worked! I dealt with database engines in the 1990s that used memmap for files tens of gigabytes in size.