Longer write-ups. Unpolished on purpose; corrections welcome.
Write-ahead logs, and why fsync lies
2026-11-02 · 12 min read
A write-ahead log gives a storage engine its durability story, and in almost every
engine that story has a seam in it. fsync() returns when the kernel has
handed the data to the device, not when the device has committed it to the medium.
On a drive with a volatile write cache, those are different events, and the gap
between them is exactly the size of the durability window you did not budget for.
The usual fix is to require a battery or capacitor-backed cache, which turns the
gap into a promise the device keeps even across power loss. The unusual fix is to
keep the WAL on a device that does not cache, which is slower and also the reason
people buy the expensive drives.
// the part that everyone gets right
write(wal_fd, entry, len);
fsync(wal_fd);
// the part that decides whether the promise holds
// (depends entirely on what the drive does with the cache)
Worth measuring on every new machine you buy: pull the power, in a lab, with data
you can afford to lose. The number you get back is the only number that matters.
Tailing a replicated log without a leader
2026-10-19 · 9 min read
Followers are cheap to read from until you need to prove a read is not stale.
There are three families of answer, and all of them trade something:
-
Counter comparison. Cheap and racy. You can read a log position,
lose the race, and still answer — the window is small but not zero.
-
Commit index reads. Correct if the leader publishes a committed
index and the follower can prove it has applied up to it. Costs a round trip.
-
Leases. Fast and correct until clocks misbehave, at which point
the failure mode is two leaders for a moment.
In practice the second option, with the read pinned to a specific log position, was
the one that survived a year of production without an incident nobody could explain.
gRPC keepalives in long-lived connections
2026-09-28 · 7 min read
The defaults assume callers that open a connection, make a call, and go away. Batch
pipelines do the opposite: one connection, hours of sparse traffic, and a NAT
somewhere in the middle with a thirty-second idle timer.
Two knobs carry most of the weight: keepalive_time on the client, and
permit_keepalive_without_calls on the server. Set the first below the
shortest idle timeout you can find in the path, and the second so the server does
not treat your keepalives as abuse.