I've been using ZFS for years and have pretty good experience with it. I've not been following Linux development too much. Is bcachefs usable/stable/reliable enough to replace it?
The author of the benchmark here. I went over some comments and I'll try to tackle them here.
I'm pretty clear that GH runner based benchmark is far from perfect due to noisy neighbours etc. Thus every test first is running so called calibration... to reject completely unreliable VMs.
I'm fully aware that this can't completely fix the issue. Can limit it but not fix.
But as of now there are 593 runs recorded so average should still be quite meaningful.<p>Having that said I'm desperately trying to get REAL hardware to run that benchmark. With some successes ;)<p>Few months ago I got Hetzner machine from Kent Overstreet and I was able to finish 3 runs before machine died...
Results: <a href="https://bartosz.fenski.pl/modern-fs-benchmark/real-hw/" rel="nofollow">https://bartosz.fenski.pl/modern-fs-benchmark/real-hw/</a><p>Currently I've got even more interesting machine with tons of disks and I'm running new set of benchmarks but it's really in its initial stage.<p><a href="https://bartosz.fenski.pl/modern-fs-benchmark/sas-hdd/" rel="nofollow">https://bartosz.fenski.pl/modern-fs-benchmark/sas-hdd/</a>
2nd run in progress... one run on REAL hardware takes much more time than on GH runner so it's slow.<p>But this new hardware has also so many disks that the plan is to try also more complex, tiered cache topologies. I'm working on it.<p>I'm happy to answer any other questions, sources of every piece of this benchmark are freely available and I'm not saying they are 100% correct. I'm open to improvements.
I went back and forth with Hetzner a couple times, I think we just got a bad machine :)<p>I've been saying it for months, but eventually I'm going to move the automated builds off the 48 core monster and we'll be able to use that for automated perf testing too. The machine we just got has spindles for EC perf testing, but the Hetzner monster has very high end enterprise ssdd.<p>Also, just got done with the Rust for Linux conference, still not home but here's slides that still need reformatting: <a href="https://evilpiepirate.org/~kent/Kangrejos-2026-bcachefs.pdf" rel="nofollow">https://evilpiepirate.org/~kent/Kangrejos-2026-bcachefs.pdf</a>
Should include:<p>1. Dual Ext4 + external 32GB journal X4 pcie SSD (the prior winner of benchmark surveys)<p>2. Bare F2FS after a trim and SSD vendor software cache flush operation (it should be slower, but knowing how much slower on identical hardware could be interesting.)<p>3. DRBD across a 48U 100Gbps host rack (single X4 pcie data drive per host, OS on primary)<p>4. CephFS across a 48U 100Gbps host rack (single X4 pcie data drive per host, OS on primary)<p>Best regards =3
> CI runs use loop devices on shared ephemeral VMs (one VM per filesystem): compare shapes and ratios, not absolute MB/s. Each job records a host-calibration anchor — see the table.<p>I think if you're not using baremetal for such tests, it's likely that the results are simply not comparable at all? What if another tenant is also using the disk?
It's a fair point but it's also possible the person running the tests has a dedicated test hypervisor for this , so that different configurations of filesystems and VMs can be created and destroyed quickly in an automated manner.<p>If it's something as simple as a KVM hypervisor that only runs 1 test VM at a time (with no other load from anything else other than the basic systemd daemons, ssh daemon etc running on the hypervisor), the results could be very close to bare metal.<p>I can see it being very time consuming and annoying to do repeated <i>manual</i> bare metal OS installs and new partitioning/filesystem creation for such a large variety of tests.<p>The author does also say that performance isn't really <i>the main thing</i> but rather, data integrity:<p><a href="https://github.com/fenio/modern-fs-benchmark" rel="nofollow">https://github.com/fenio/modern-fs-benchmark</a>
> I can see it being very time consuming and annoying to do repeated <i>manual</i> bare metal OS installs.<p>Well don't do that then. There's lots of other options. Probably the simplest is a single bare metal install on a simple filesystem on one device. run the filesystems under test on other storage dedicated to testing.<p>You could also boot into a network install and use local storage exclusively for testing.
For the main linked benchmark there are no dedicated test hypervisors. It's all based on GH runners with all the limitations and quirks that came with it.<p>Real hardware is used in:
<a href="https://bartosz.fenski.pl/modern-fs-benchmark/real-hw/" rel="nofollow">https://bartosz.fenski.pl/modern-fs-benchmark/real-hw/</a>
<a href="https://bartosz.fenski.pl/modern-fs-benchmark/sas-hdd/" rel="nofollow">https://bartosz.fenski.pl/modern-fs-benchmark/sas-hdd/</a><p>But unfortunatelly it's much more limited number of actual runs.
sas-hdd is still in progress so numbers for it should increase over time.
> compare shapes and ratios, not absolute MB/s<p>In this case, given that the author's own disclaimer (above) already disclaims the numeric readings, I'm not sure how it's possible to make any inference on "shapes and ratios" derived from the numeric readings.
Seeing great results from bcachefs just makes me more sad that Kent and the other kernel devs couldn't come to an understanding to keep bcachefs in-tree. I want to use it for my storage arrays so badly, but I'm still stuck with btrfs as the only available in-tree filesystem with modern features.
So two filesystems that are essentially shunned from the Linux kernel and permanent second-class citizens, and one that was removed from Red Hat and has a questionable history of reliability. Oh boy which do I choose?<p>I'm saying ZFS on another OS.
no thanks to red hat. i switched to debian for servers. fedora fortunately still supports btrfs.<p>zfs is shunned. can't really get around that copyright issue. bcachefs just hit a setback, which i am hopeful will eventually be resolved.
xfs/lvm-raid10 or ext4/lvm-raid10, obviously.
If I'm forced to use Linux, sure.<p>I could drop bricks on and cord pull those all day and they would not lose data. Which is a small ask for a filesystem IMO.
would want to have compression..
BCacheFS is the best Linux filesystem now that storage costs a premium.<p>You can mix devices of different sizes and types on bcachefs. You can have foreground and background devices to balance performance and also different compression settings for foreground and background transactions.<p>You can set replicas=N to the individual file or directory on bcachefs. For example files you can just re-download or re-build. Likewise you can set a higher number of copies to important files.
What is md-raid10 doing that is so much worse than lvm-raid10? In terms of "I/O" and "responsiveness." It's not really obvious to me from either the linked page or <a href="https://github.com/fenio/modern-fs-benchmark" rel="nofollow">https://github.com/fenio/modern-fs-benchmark</a> . In principle they should be similar?
> Every push/2-hourly cron builds each filesystem across 4 loop devices backed by sparse files, runs the suite, and publishes a results table in the job summary plus JSON artifacts.<p>I get that real hardware costs (author mentions EUR 70 a month for a suitable server), but without at least a baseline snapshot comparison run between real hardware, both SSD and HDD, and the sparse file-backed loop devices, it's hard to take much away from this.<p>Sadly the AI apocalypse isn't making stuff like this easy to do as a hobby.
<a href="https://github.com/fenio/modern-fs-benchmark" rel="nofollow">https://github.com/fenio/modern-fs-benchmark</a>
Oh, so bcachefs is doing pretty well.
what does "integrity" fail mean in the first table? that the case didn't recover from the 2G corruption?
For peace of mind I'm still using zfs (since the last 15+ years) but I'm definitely not impressed by the performance...
Different tools for different jobs; use ZFS for your data store and ext4 for your primary drive.
Depending on your current setup, dRaid can give you quite the performance boost nowadays, since its data slabs are spread out over all disks in the array its using all of the disks bandwith combined.
zfs-mirror-8k (whatever that is) looks alright.
The script[1] shows it's a ZFS mirror (RAID-1) with 8k max record size, as opposed to the default 128k.<p>I say max, as ZFS stores multiples of the physical block size up to the record size. So even with 128k record size, if your file fits in two physical blocks, it will write two physical blocks of data.<p>The `ashift` parameter[2] controls the physical block size, typically 512 bytes or 4k for HDDs. Though higher can be useful on SSDs (but less tested, have seen some bug reports with >4k block sizes).<p>Typically you'd only want to use such small records if you are tuning it for a specific workload, like if you have a database that writes 8k pages.<p>Compression acts on records, so by limiting the record size, you limit the effectiveness of the compression: 1.2 and 1.8 blocks worth of data both gets written as 2 physical blocks.<p>On the upside, record size is a dataset property, so you can have many different datasets with different record sizes on the same pool.<p>[1]: <a href="https://github.com/fenio/modern-fs-benchmark/blob/599ec72fe335e2c043d83daa69c870fcf9fa8e72/scripts/fs/zfs.sh#L34" rel="nofollow">https://github.com/fenio/modern-fs-benchmark/blob/599ec72fe3...</a><p>[2]: <a href="https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Workload%20Tuning.html#alignment-shift-ashift" rel="nofollow">https://openzfs.github.io/openzfs-docs/Performance%20and%20T...</a>
It could use some explanation for the exact setups. Like how many drives are there in each raid, what does the ec config actually look like, etc. As far as I can see it's not described.
I think the reviews should also include the social aspect of these filesystems...<p>There is and have been many promising and exciting FS to replace the old boring ones, but for storage you not only want to avoid technical issues but also maintainer(s) drama...
Why do people keep bringing up drama?<p>The community infighting has sucked, but that's a thing that matters primarily for maintainers.<p>I think most users just want something that works.
drama matters because most users don't want to have their favourite FS randomly removed from the kernel unexpectedly after some OS update :)<p>That said I certainly hope that one day the technical advantage of bcachefs will be so overwhelming that maybe the decision to remove it will be overturned. And if big vendors make it their default FS the bus factor will disappear (even if unofficially you'd still be the sole maintainer, but no one cares about that in the enterprise world...)
I don't care about the drama, I just want to say thank you for your continued work to advance the state of the art in open file systems.
Related username?<p>To answer the original question, most people who care about their filesystem at all care about its stability. Not just "does it work now" but also "will it work and improve over time". Infighting puts the future at risk.
It really does.<p>But you might want to check out the bus factor on btrfs too; when a maintainer says "but we've saved Facebook billions and billions of dollars!", calls for the other filesystem maintainer to be ejected from the community, then quits to join Anthropic a month later - that's not a vote of confidence.<p>I'd be very happy if people could just stop bringing up drama and us factors. We put it behind us a year ago, but it seems not everyone got the memo.
Reiserfs was the OG drama fs
That's what I meant, it's always risky to rely on a single person for such an important subsystem as storage... I'm also very sad that bcachefs has been kicked out of the kernel, it is really a very promising FS.<p>So that leaves btrfs which always seemed complex and brittle to me (compared to zfs at least)
xfs is by far the slowest in "ftruncate empty file to 1G" but the fastest in "ftruncate 256M file to 512M" - does anyone know why?
@farlight assuming that you're the creator do you think you'd be able to rework the HTML/CSS? I'm sure you've got good data but speaking on behalf of my eyeballs, the results page is... hard to read!
> 2G of random garbage is written directly onto one member device (behind the filesystem's back, offset 1G — python injector; uutils dd mis-seeks on dm devices), caches dropped, then a full scrub: btrfs scrub -B, zpool scrub + wait, bcachefs scrub, md/lvm sync-action 'check' (which can only COUNT mismatches — no checksums to know which copy is right).<p>I'm not sure that nuking 2G of the underlying block device is a recoverable error on any filesystem that I'm aware of? Can you confirm if any ofthe filesystems really came out of the other side in a usable state after scrubbing?<p>-----<p>> Trivial-op p99, idle (ms) # A trivial operation — one 4k write + fsync every 200ms (like a shell appending history or an editor updating its swap file) — run alone for 10s. p99 of the fsync completion<p>In fact, if it's OK for me to ask, are any of the metrics tht you used standard industry metrics? It looks like several of the tests are bypassing the kernel's page cache? -- which I worry may fall into the trap of "I modified the system to be unrepresentative of reality and then tested it".<p>----<p>> kernel 7.0.0-1012-azure<p>Can you confirm if you tested on a bare metal machine? were you the only tenant?
> I'm not sure that nuking 2G of the underlying block device is a recoverable error on any filesystem that I'm aware of?<p>ZFS and btrfs were designed from the start to handle this, by using checksums on every piece of (meta)data and redundancy to return the same data as was stored to the kernel, and rewrite the bad data.<p>I've tested my own machines running ZFS by random writes out of band from the filesystem/kernel and it has always found and fixed them.
1 device out of the replica set I’m assuming so all of them should recover.
The integrity check is only on the tests which are either RAID or the filesystem equivalent.
what exactly would you like to improve?
I'd actually recommend some excellent books on the "philosophy" of data presentation: The first that comes to mind is "the visual display of quantitative information" by Edward Tufte seems to be freely available online, and the other one on my mind is "how charts lie" by Alberto Cairo (which doesn't seem to be freely accessible)<p>But if it helps, just some "initial gut feel observations" from me:<p>* It's definitely not possible to find issue with the the _sheer amount_ of results, but there's just far too much for a human to absorb, all presented at once<p>* Overall text size is quite small, and difficult to read<p>* The page doesn't make a strong statement of _what_ is under test: the first words are: "modern-fs-benchmark Multi-device CoW filesystems under workloads classic benchmarks skip" -- which defines the webpage in terms of what it is _not_, without stating what benchmarks are actually present.<p>* The first line of teh page contains run statistics that probably eithre want to b at the bottom, or just don't need to be in the webpage at all: "latest run 2026-09-18 18:50:45 UTC, kernel 7.0.0-1012-azure, 593 runs recorded · 145 trend points shown"<p>* A significant proportion of the free text is caveats. There's nothing wrong with being transparent about limitations, but they may be a sign that there might be alternative ways to present the data, or that the data may be flawed (depending on the caveat)<p>* Theres several categories that I think have been invented for the purpose of collation, but I don't think are defined on the page. I think "Overall Core" and "Core I/O" aren't explained, which means by definition it's impossible for a reader to understand the score table.<p>* And as we're all aware right now, current Claude models are currently struggling to write coherent English. There's several incoherent sentences on the page. It's a Claude issue.
Speaking as someone who consumes this, I appreciate how it's laid out. As a developer, we can often see at a glance where the bottleneck is if we have enough data laid out - IOW, data overload for you is me feeling like a kid in a candy store.<p>Sometimes there are ways to make things easier without dumbing them down, but way too many people conflate the two; I get nervous when non engineers say "I've studied this, it should be easy".
It'll be interesting when people start applying similar effort that is going into LLM search for security vulnerabilities to finding conventional reliability/performance bugs.
clicked, ctrl+f ssd, hdd, not found, closed