Open the same Develosaur board in two browser tabs, and something you would never notice becomes possible: one tab can die mid-write and leave the other tab depending on bytes that never reached the server.
That is relay poison, the name we gave to a four-bug chain our chaos rig found before a real board ever ran into it. Here is the mechanism, the fix, and why we now kill tabs on purpose in CI.
The setup: sharing one document before it is durable
Develosaur's canvas runs on a Yjs CRDT document, one per project. When one of your tabs makes an edit, it does two things at once: it appends the update to a local outbox (a durability buffer that survives a reload before the row is safely written to Postgres) and it relays the raw update to any sibling tab open on the same board, over the browser's BroadcastChannel API, so the second tab's canvas updates instantly without waiting on a round trip to the server.
That relay is why two tabs feel like one document. It is also the crack relay poison lives in.
The gap: a sibling depends on bytes that were never durable
A Yjs update can reference an earlier update as its origin, the way a CRDT tracks causal order. If Tab A pushes an edit and relays it to Tab B, and Tab B's next edit is causally built on top of A's, then B now depends on a piece of state that, from the server's point of view, does not exist yet. A's outbox append is still in flight.
Kill Tab A right there, before that append lands, and the dependency never resolves. Build a fresh copy of the document from what actually reached the server, and it holds pendingStructs: a queue of updates waiting on a clock that will never arrive. The document does not error. It just quietly stops being complete, and nothing in the UI tells you that.
We found this the way we find most of the ugly ones: not in production, but on the torture rig, an internal harness that runs several writers against one board and injects failures on purpose, killing tabs mid-write, dropping connections, racing agents against each other. A cold load off the resulting Postgres rows kept parking on clocks that had never reached the server. That was the first clue that something upstream of storage, not storage itself, was leaking updates that depended on other updates that were never coming.
The fix: give the sibling something to adopt
Relay poison only has one way out: a live sibling has to be able to stand in for a dead tab's unflushed write. Two changes make that possible, and neither touches the server protocol.
First, the relay message had to stop being anonymous. It used to carry just raw update bytes; now it carries the outbox row id that the sending tab minted the moment it captured the edit, before the append even started. A sibling receiving the relay can now tell exactly which row those bytes belong to.
Second, on receipt, a sibling mirrors that row into its own local outbox under the ORIGIN tab's id, not its own, and only if no row with that id exists yet. Nothing changes if the origin tab is fine: it flushes its own row and the mirror sits there unused. But if the origin dies, its hold on that row (a producer lock scoped to that tab) disappears, and the orphan sweep that already exists to recover a dead tab's own unflushed rows finds the mirrored copy sitting under the origin's id and flushes it, typically within five seconds.
We did not build a new recovery path. We fed an existing one a copy of the row it was missing.
One guard on top: a sibling will not resurrect a row that has already been acknowledged and flushed, even though doing so would be harmless to the CRDT itself (duplicate rows are the accepted cost of this design, not a correctness bug). It is log hygiene, kept as a short list of recently-acked ids in memory, nothing more.
Proving it, not assuming it
A fix for a race condition is worth nothing until you try to lose the race again. Closing this one meant:
- rebirthing the test project's document from scratch and rerunning the exact scenario that first parked it: green
- a chaos run that kills tabs mid-write for ten minutes straight: green
- a scan across every row for clock gaps, the same signature that first surfaced the bug: clean
- the push-triggered CI torture workflow, which had been red on this exact failure twice: green
What it costs, and what it buys
The cost is small and constant: one mirrored row per open sibling tab, most of which are never used. What it buys is a guarantee we would rather not have to explain after the fact: a tab dying at the worst possible moment does not leave a hole in your document that nothing will ever fill.
This is the same durability line the priority engine and task claims run on, and the same CRDT sync described in Agentic Workflow MCP. If you run several tabs, or several agents, against one board, this is the kind of bug we go looking for before you ever have to.
More engineering notes like this live on the blog.