KVStore is a key-value database I built from scratch in Go — no Redis, no RocksDB, no embedded database, just the standard library — and the one feature the whole project exists to prove is durability: once the server says a write succeeded, that write has to survive a kill -9, a torn log segment, or the machine losing power mid-fsync. This is a tour of the write-ahead log that makes that promise good, and the test suite that tries hard to prove it wrong.

Why "the write succeeded" is a hard sentence to earn

An in-memory map is fast and it is also a lie the moment the process dies. KVStore's storage engine is a sharded map — per-shard RWMutex, so reads and writes on different shards never block each other — and on its own that map is exactly as durable as the process holding it, which is to say not at all.

The fix everyone reaches for is a write-ahead log: before the server acknowledges a write, it has to be on disk somewhere it can be replayed from. That's a one-sentence idea with a lot of ways to get wrong — the log itself has to survive the same crash the in-memory state doesn't, which means thinking about partial writes, fsync timing, and what "on disk" actually guarantees on the filesystem you're running on.

Group commit, CRC32C, and three fsync policies

Every WAL record carries a CRC32C checksum, and records are written into rotating segment files rather than one ever-growing log — that bounds how much a single corrupt segment can cost you, and it's what lets recovery reason about "this segment" instead of "the log, somewhere."

The part that actually costs performance is fsync, so KVStore batches concurrent writers into group commits: instead of one fsync per write, several writers that land in the same short window share a single fsync call. That turns a design question — how durable do you need this write to be — into a runtime flag instead of a rewrite: --fsync always syncs every record immediately (safest, slowest), --fsync everysec syncs on a timer (the usual compromise, bounded data loss on a crash), and --fsync no leaves it to the OS (fastest, no durability guarantee at all). Same knob either as a flag or an environment variable — KV_FSYNC, following the project's own convention that every flag has an env equivalent and an explicit flag always wins.

Recovery: a torn tail is not the same as corruption

On restart, the engine loads the most recent snapshot and replays the WAL segments written since it — and that replay has to make a judgment call every single time: is this broken record at the end of the log a torn tail (the process died mid-write, exactly as expected, truncate it and move on) or real corruption (a checksum mismatch somewhere in the middle, which is a bug and should fail loudly rather than silently drop data)? Getting that distinction wrong in either direction is bad — refuse to start on every ordinary crash, or silently eat a corrupted key you should have flagged.

Snapshots get the same care. Writing one is atomic in the boring, correct way: write to a temp file, fsync it, rename it into place, then fsync the directory so the rename itself is durable, with a CRC32C footer over the whole file so a half-written snapshot can never be mistaken for a good one. None of this is exotic — it's the standard recipe — but every step in that recipe is a step someone skipped in a database that later lost data.

Proving it: kill -9, for real

You can't SIGKILL a goroutine, so the crash-test suite spawns real, separate OS processes running the actual server binary. A writer records every key the server said "OK" to, the process gets SIGKILLed at a random moment (200–999ms in, with writers still active), and after restart every single acknowledged key has to be present and correct — make test-crash runs this across two fsync policies, always and everysec, twelve iterations each: 24 kills in total, timed randomly rather than targeted at a specific point in the write path.

That suite doesn't stand alone. A differential test runs random operation sequences against both the real store and a twenty-line reference map and compares every result, automatically shrinking any failing sequence down to a minimal reproduction. A chaos suite throws forged length fields, mid-handshake TLS bytes on the plain KV port, and clients that disconnect mid-frame at the server, then checks a well-behaved client still gets correct service afterward. And the protocol decoder is fuzzed directly, asserting it never panics and never reads past its own input. Four different ways of trying to be wrong, because "the tests pass" and "this is actually durable" are not the same claim.

One honest gap: directory fsync on Windows

On Linux and macOS, "fsync the directory" is a real syscall — fsync(2) on the directory's own file handle — and that's what makes a brand-new file's directory entry durable, not just its contents. Windows has no equivalent, so that specific guarantee is weaker there: record contents are still fsynced under --fsync always, but the durability of a new file's directory entry is not. This is documented rather than hidden — it's written up as its own decision record — and it's visible at runtime too: every STATS response reports a wal.dir_sync_supported field, so a replica or a monitoring script can see which guarantee it's actually getting instead of assuming.

What I'd carry into the next project

Treat durability as a testing problem, not just an implementation one. It's easy to write a WAL that looks correct and never find out it isn't until a real crash loses real data — SIGKILLing actual processes, not mocking the failure, is what turns "should be durable" into something you can actually defend. The other habit worth keeping: protocol, store and wal have no knowledge of each other, so recovery logic can't accidentally depend on something the protocol layer assumed. One package composes them; nothing else is allowed to.

Source: github.com/raqueeeb/kvstore · Case study: raqueeb.dev/projects/kvstore · Portfolio: raqueeb.dev · GitHub: @raqueeeb · LinkedIn: in/raqueeeb