At one point we were working out how we wanted to store the PowerMTA spool for a busy mail platform.
The requirement sounded simple enough. The spool gets hammered when the system is busy, so the storage underneath it needed to be fast, resilient and as boring as possible.
One of the first things I seriously looked at was GlusterFS.
I liked it on paper. We wanted replicated storage across physical machines, Gluster was built for distributed storage, and we had proper dedicated hardware in the lab with fast NVMe underneath it. I spent a good few weeks building the cluster, tuning it and getting it into a state where I was happy to start throwing realistic workloads at it.
That bit is important: Gluster never went into production. The whole reason we had the lab was to find this stuff out before it did.
What the workload actually looked like
PowerMTA stores messages in spool files while they are queued for delivery. Its own documentation even recommends spreading spool directories across multiple physical disks when more performance is needed, specifically to distribute the I/O load.
Our underlying filesystem was XFS on NVMe, so without any distributed storage in the middle the path was pleasantly boring.
The important thing is that a mail spool is not a nice sequential workload. It is a huge number of small files being created, opened, read, updated and eventually recycled or removed as messages move through delivery.
PowerMTA has settings such as sync-msg-create and sync-msg-update because receiving a message and later updating its delivery state can both involve synchronising changes to disk. Its documentation gives marking recipients as handled as one example of an update. That should give you a decent idea of how active those spool files can be.
When we started simulating 500,000 to one million emails an hour, the Gluster volume began getting buried in I/O. That still was not particularly close to the peak throughput we expected the finished platform to handle. Yeah, we sent stuff pretty fast.
A million messages an hour works out at roughly 278 new messages entering the spool every second. That is before you count delivery reads, state updates, retries and files leaving the queue.
The email bodies themselves were usually tiny, so this was not really a question of whether the NVMe could sustain enough megabytes per second. The problem was the number of filesystem operations surrounding all of those little files.
Gluster’s own performance testing documentation makes the same distinction. For small-file workloads, it points out that performance is dominated by opening, closing and metadata operations rather than simply reading or writing the contents of the file. It even suggests measuring that sort of workload in files per second rather than MB/s.
That was almost exactly what our PowerMTA spool had turned into.
It looked good until we started hitting it properly
I wish I still had the numbers from those tests because I spent enough time staring at them. I don’t, so I’m not going to pretend I remember exact IOPS or latency figures years later.
What I do remember very clearly is what happened once the tests started getting close to the sort of load we actually cared about: the storage layer became noticeable.
The hardware underneath it wasn’t the problem. We’d deliberately built the test stack on fast local storage because we knew the spool was going to be demanding. Gluster worked, but once we started pushing hundreds of thousands of tiny mail files through it, the storage path just became saturated with I/O.
The difference from local XFS was that every filesystem operation now had considerably more machinery behind it.
Gluster’s architecture explains how requests travel through FUSE and its translator stack, while AFR handles replication across the bricks. That abstraction is useful when you actually need a distributed filesystem. In our case it meant that hundreds of tiny local filesystem operations had become distributed filesystem operations.
So I did what you normally do. Tuned it, changed things, tested again, assumed I’d configured something badly, found something else to tweak and ran the load again.
We changed caches, Gluster settings, networking and anything else that looked remotely relevant. It got better in places, but the underlying behaviour never really changed. Once enough little spool files were in flight, we were spending far too much time fighting the storage layer.
After a while the more useful question became: why am I trying this hard to make Gluster fit the workload?
Gluster wasn’t broken. It was just doing more than we needed. A replicated Gluster volume has a distributed filesystem, replication and its own I/O path sitting between the application and the storage underneath it. That’s the point of it. Gluster’s replication documentation explains that part properly if you want the details.
For this job, though, we didn’t really need a clever distributed filesystem. We needed very fast storage with another copy of the data somewhere else if a machine died.
That pushed me towards DRBD.
DRBD made much more sense for the way we already worked
We already used Linux software RAID heavily and were comfortable with the normal block-device stack, so DRBD felt much less invasive.
Instead of giving us another filesystem abstraction, DRBD works down at the block-device layer and replicates blocks between machines. The DRBD user guide documents using normal Linux block devices underneath it.
That meant the spool could go back to behaving like a normal local XFS filesystem. PowerMTA still opened, read and updated its tiny files through the ordinary Linux filesystem path. DRBD dealt with replication underneath that filesystem instead of turning each filesystem operation into a distributed one.
And when we ran the same kind of load against it in the lab, the difference was obvious enough that I stopped caring about trying to rescue the Gluster design.
DRBD basically disappeared from the performance side of the conversation.
Obviously replication isn’t free. If you’re doing synchronous replication, the remote write has to complete before the local write can be considered done. But for our workload, that cost wasn’t turning into the problem we’d been chasing with Gluster. The storage behaved much more like local storage again.
I can’t give you a nice benchmark graph showing DRBD winning by some suspiciously precise percentage. I no longer have the test data, and even if I did it would only describe our hardware, our configuration and our workload.
What mattered at the time was much simpler: with Gluster we were spending time investigating the storage layer under load; with DRBD we weren’t.
So that was the design we took forward.
Fast doesn’t mean you can ignore the failure cases
DRBD has its own ways of ruining your day if you design it badly. Split brain is real, and replicated block storage is not something you throw together and hope sorts itself out during a network partition.
We treated failure handling as part of the design from the beginning. DRBD has quorum support, but simply having multiple machines doesn’t magically solve the problem. You still need to think about which side is allowed to continue, promotion, recovery and what happens when two machines disagree about the state of the world.
That wasn’t a reason not to use it. It was just part of building the service properly.
What I liked was that outside of the replication layer, the storage still looked like normal Linux storage. We weren’t carrying a completely different distributed-filesystem model just for one workload, and we kept control over the lower layers rather than handing the whole shape of the stack to Gluster.
GlusterFS wasn’t bad. If I needed the distributed filesystem Gluster provides, I’d judge it against that problem. We just discovered in the lab that it wasn’t the problem we actually had.
I’m glad we discovered that there rather than after putting it into production.
I’d already spent weeks building and tuning the thing, and it would have been very easy to keep going because of the time I’d sunk into it. Instead we stopped, tested a simpler design and went with the thing that behaved better under the workload that actually mattered.
We didn’t replace GlusterFS in production. We did the much cheaper thing: we proved it wasn’t the right fit before it ever got there.
