Notes

Longer write-ups. Unpolished on purpose; corrections welcome.


Write-ahead logs, and why fsync lies

2026-11-02 · 12 min read

A write-ahead log gives a storage engine its durability story, and in almost every engine that story has a seam in it. fsync() returns when the kernel has handed the data to the device, not when the device has committed it to the medium. On a drive with a volatile write cache, those are different events, and the gap between them is exactly the size of the durability window you did not budget for.

The usual fix is to require a battery or capacitor-backed cache, which turns the gap into a promise the device keeps even across power loss. The unusual fix is to keep the WAL on a device that does not cache, which is slower and also the reason people buy the expensive drives.

// the part that everyone gets right
write(wal_fd, entry, len);
fsync(wal_fd);

// the part that decides whether the promise holds
// (depends entirely on what the drive does with the cache)

Worth measuring on every new machine you buy: pull the power, in a lab, with data you can afford to lose. The number you get back is the only number that matters.


Tailing a replicated log without a leader

2026-10-19 · 9 min read

Followers are cheap to read from until you need to prove a read is not stale. There are three families of answer, and all of them trade something:

In practice the second option, with the read pinned to a specific log position, was the one that survived a year of production without an incident nobody could explain.


gRPC keepalives in long-lived connections

2026-09-28 · 7 min read

The defaults assume callers that open a connection, make a call, and go away. Batch pipelines do the opposite: one connection, hours of sparse traffic, and a NAT somewhere in the middle with a thirty-second idle timer.

Two knobs carry most of the weight: keepalive_time on the client, and permit_keepalive_without_calls on the server. Set the first below the shortest idle timeout you can find in the path, and the second so the server does not treat your keepalives as abuse.