DDIA Chapter 5: Replication — How ClawMetry Keeps Cloud Reads Fresh

Chapter 5 of Designing Data-Intensive Applications is called "Replication." Martin Kleppmann opens it with what sounds like a simple observation: keeping a copy of the same data on multiple machines is hard because the data keeps changing. The rest of the chapter is a methodical tour through why that's hard and how different systems handle it.

I reread it recently after we shipped a fix for a production incident. A first-run ingest of 465 sessions blocked the ClawMetry cloud relay's heartbeat for nearly two and a half minutes. During that window, every Brain event read and transcript load from the cloud relay queued up and never arrived — browsers gave up and users saw spinning loaders where their live agent data should have been.

The fix took one line. Understanding why it was the right fix took rereading Chapter 5.

Our replication topology

ClawMetry isn't a traditional distributed database, but it runs a replication topology that DDIA Chapter 5 maps onto cleanly.

The sync daemon (clawmetry/sync.py) is the leader. It is the only process that holds the DuckDB write lock. It ingests three data sources — JSONL session files from ~/.openclaw/, live events from the OpenClaw gateway WebSocket, and optional OTLP traces — and writes everything to a local DuckDB store. The daemon is also responsible for pushing an end-to-end encrypted snapshot to ingest.clawmetry.com.

ClawMetry Cloud is the follower. It receives AES-256-GCM encrypted snapshots from the daemon and holds them until a browser asks for live data. The browser decrypts client-side — the cloud relay never sees plaintext.

The dashboard and API route handlers are read-only consumers. They read from DuckDB through a lightweight localhost proxy (clawmetry/local_server.py) so they never compete for the write lock. For the cloud-hosted version, they read through the relay instead.

In DDIA terms: one leader, two classes of followers (the cloud relay and the local query server), and asynchronous replication from leader to cloud via periodic encrypted snapshots.

Replication lag — the hidden cost of async

Kleppmann is clear on the core tradeoff of asynchronous replication: the leader doesn't wait for the follower to confirm receipt before acknowledging a write. This is fast and resilient — the leader keeps processing even if the follower is temporarily unreachable. The cost is replication lag: for some window of time, the follower is behind the leader.

In ClawMetry's design, that lag window is defined by the heartbeat interval. The daemon runs a main loop. At the end of each iteration, it calls the heartbeat: it builds the encrypted snapshot, pushes it to ingest.clawmetry.com, and drains any pending_queries that the cloud relay is holding — Brain time-window fetches, transcript reads, device approval requests. Pending queries don't resolve until the heartbeat fires.

Under normal conditions, a main-loop iteration completes in a few seconds and the heartbeat fires promptly. Replication lag stays low. A user loading the Brain stream from the cloud sees events that are a few seconds stale at most — acceptable for an observability dashboard.

The incident: 465 sessions, 2 minutes 40 seconds of lag

On 2026-07-30, we got reports from a new Cloud user that their Brain stream was showing nothing. Transcripts weren't loading either. The user had just installed ClawMetry for the first time on a machine with a history of 465 sessions accumulated across months of OpenClaw usage.

The first-run ingest path processes all existing sessions to populate DuckDB. We iterate through every JSONL file, parse events, and insert them. At the time, this loop ran to completion before the daemon proceeded to the heartbeat.

On 465 sessions, that loop took approximately two minutes and forty seconds.

For two minutes and forty seconds, the heartbeat never fired. Every pending query the relay was holding — the Brain window load the new user had triggered by opening the dashboard, plus subsequent transcript requests — sat queued with no response. The browser's request timeout is shorter than 160 seconds. Everything showed up empty, and the user assumed ClawMetry wasn't working.

This is Kleppmann's replication lag problem, materialized: we had a leader that was too busy writing to replicate. The follower's state was frozen. Reads on the follower returned stale data — in this case, stale enough that the data effectively didn't exist yet from the browser's perspective.

The Brain event stream during a first-run ingest: nothing arrives for the full backfill window, then events appear all at once when the first heartbeat fires

The fix: keepalive heartbeats inside long ingest passes

The fix was adding a call to _ingest_keepalive_heartbeat(config) between each session processed during the first-run backfill (and any other ingest pass that could run long). Instead of running the full 465-session loop and then heartbeating, the daemon heartbeats periodically throughout.

for session_path in sessions_to_backfill:
    _ingest_session(session_path, conn)
    # yield to the cloud relay between each session
    _ingest_keepalive_heartbeat(config)

The keepalive function is lightweight: it checks whether the last heartbeat is older than the configured threshold, and if so, fires one immediately. For a 465-session backfill, this means the relay gets a heartbeat after every session — worst-case lag is the time to process one session (typically a fraction of a second), not the time to process all of them.

After the fix, a new user with 465 existing sessions sees their Brain stream within a few seconds of opening the dashboard. The backfill still runs, but it no longer starves the relay.

What DDIA Chapter 5 names that we experienced

Reading the chapter after the incident clarified the vocabulary for what we built and broke.

Read-your-writes consistency

Kleppmann describes a guarantee where after a user writes something, they should always be able to read what they just wrote. We violated this: the user who triggered a first-run ingest (the write) couldn't read any of their agent data on the cloud relay (the read) until the backfill completed. The user had made a write that the follower hadn't processed yet, and reads from the follower returned nothing.

Monotonic reads

A softer guarantee: a user shouldn't see data go backwards in time. During the backfill window, the brain stream showed nothing, then suddenly populated when the first heartbeat fired. That's not going backwards, but it's disorienting in the same way — the user refreshed the browser multiple times seeing nothing, then saw everything on a later refresh. The "moving target" experience Kleppmann warns about.

Asynchronous replication and durability

Chapter 5 is careful about durability: if the leader fails before the follower has received the latest data, that data is lost from the follower's perspective. For ClawMetry, our durability story is that the leader (your local machine) is always the system of record. The cloud relay is a secondary view, and if the daemon is unreachable for a few minutes, the cloud relay shows a stale snapshot. This is a deliberate tradeoff: we don't write to the cloud first and then to local DuckDB. Local-first means the cloud is eventually consistent, not strongly consistent.

This maps to DDIA's "asynchronous replication with potential data loss on leader failure." In practice, for an observability system, this is acceptable: if your machine is down, your agents aren't running either. The cloud showing a stale snapshot is correct behavior.

The rule we codified

The incident prompted a convention we added to the codebase:

Any ingest pass that can run long must call _ingest_keepalive_heartbeat(config) between items, or every hosted relay read sits on relay_pending until the browser gives up.

This is the practical application of what DDIA Chapter 5 argues in theory: replication lag is a product of how much work the leader does between replication events. The only way to bound lag is to bound the leader's work between heartbeats. Batch sizes, iteration counts, and loop depths all have replication implications, even in a system where "replication" is just an encrypted HTTP POST to a relay.

If you're building any system with a background sync loop — not just ClawMetry — this is the question to ask: what is the longest possible gap between replication events, and what is the user-visible consequence of that gap? For us, 160 seconds of lag turns a dashboard into a blank screen. Knowing the number forces you to bound it.

What we considered and didn't do

DDIA Chapter 5 covers multi-leader and leaderless replication as alternatives. We considered them briefly.

Multi-leader (letting the cloud relay accept writes) would let the dashboard work even if the daemon is busy ingesting. But it introduces write conflicts: what wins if the daemon and the relay both write different data for the same session? For ClawMetry, your machine is always authoritative. The cloud relay has no ground truth. Multi-leader isn't an option here.

Leaderless replication (quorum writes/reads, like Dynamo) requires multiple independent replicas. We have exactly two points in our topology: your machine and the cloud. Quorum of two means both must agree, which is worse than a single leader in practice.

We stayed with leader-follower and fixed the lag instead. The right fix for replication lag is almost always to reduce the leader's work between replication events, not to change the topology.

The meta-lesson

Chapter 5's underlying argument is that replication is never free, and the cost isn't paid at write time — it's paid at read time, by users who see stale data and don't know why. The lag is invisible until someone tries to read during a wide replication window and gets nothing back.

For observability systems specifically, this matters more than for most. An observability tool that shows stale data during the exact moments when something is going wrong is failing at its core job. If a new user's first experience is a blank brain stream for 2.5 minutes while their agents are actively running, the tool has failed — not technically, but from the user's perspective.

The fix cost one line of code. Understanding why it was necessary — and what the broader class of failure looks like — cost rereading Chapter 5. Both were worth it.