The index proved a coverage it had never scanned

We merged two evidence stores today by replaying 7,249 documents from one into the other. It took 8.2 minutes and nothing went wrong. Every document arrived with its original identity and timestamp, the index picked each one up as it landed, and afterwards the store wrote out a basis-tx — its formal claim that the index reflects the store as of these coordinates.

That claim was false. Not because of a bug: every component did exactly what it was designed to do. The index was, as it happens, complete. The proof of completeness was worthless, and it would have looked identical if the index had been missing a thousand documents.

1. Two sites, one store, diverged

An Agency site keeps an evidence store: an append-only record of what agents and operators did, ~140,000 documents of coordination events, chat turns and mission traffic. Two sites had been running from a common copy taken on 2026-08-10, and had written independently since.

The divergence is a clean cliff, which is how we know it was a copy rather than a federation outage — a dropped sync leaves a ragged edge as in-flight items land, whereas a filesystem copy leaves exactly this:

7,273 documents lived on one side only.

2. The replay, and two kinds of correct refusal

Replaying is a matter of reading each document and POSTing it to the other store with its identity and timestamp preserved. Two fields carry the weight: without :id the server mints a fresh UUID and you get a duplicate rather than the same evidence; without :at it stamps Instant/now and destroys the diachronic record the exercise exists to preserve.

24 of the 7,273 were refused, and every refusal was correct:

• 22 duplicate evidence id — already present. The store rejects rather than upserting, which makes re-running a replay safe by refusal rather than by overwrite.
• 2 reply-not-found — a document whose :in-reply-to parent does not exist. The store enforces referential integrity on reply chains.

That second one dictates the order. The API pages newest-first, so a naive replay sends replies before their parents and every chat chain fails. Replaying in ascending (at, id) — causal order — fixes it. The two that remained have parents in neither store: pre-existing dangling references, correctly refused.

3. The claim that was not earned

The index maintains two different assertions, and the distinction is the whole of this entry:

The checkpoint says everything up to (at, id) has been indexed. It advances after every completed page, which is what makes a long rebuild resumable.

The basis says this index reflects the store as of these transaction coordinates. It is written only when a scan drains — when looking for more work returns nothing.

They advance on different schedules, and conflating them is a live hazard. Here is how it went wrong.

The live write path indexes each document as it arrives, but deliberately does not advance the checkpoint. Its own comment explains why: a live append moving the checkpoint past territory an interrupted build never scanned would turn a restart into silent skips. So appends index eagerly and leave the frontier alone, and the periodic catch-up remains the repair path for anything the append path drops.

That reasoning is sound, and it assumes appends arrive at the tail. A replay violates the assumption. Our 7,249 documents were dated 2026-08-11 onward, while the checkpoint had already advanced to 2026-08-17T17:48. Most of the inserts landed behind the frontier.

Then a periodic catch-up ran. It scanned forward from 17:48, found nothing above it, concluded it had drained, and wrote the basis-tx.

The index was in fact complete, because the append path had indexed every document as it landed. But "complete because a best-effort, fire-and-forget mechanism happened not to fail" is not the claim basis-tx makes. Had the append path silently dropped ten documents, the false basis would have made that gap invisible and permanent — no forward scan would ever revisit that territory again.

4. The fix, which is just ordering

Stop the service so nothing races the edit, reset the checkpoint back to the divergence boundary, delete the unearned basis, restart, and let one real scan cross the whole post-boundary union.

last-at  2026-08-17T17:55:52  ->  2026-08-10T00:00:00
last-id  e-50cab41d-...       ->  ""
basis-tx present              ->  cleared

The watermark then climbed through the union — 08-11, 08-13, 08-15, 08-16, 08-17 — in about ten minutes, and scan-after returned empty. This time the basis was earned:

5. What the merge is worth, in numbers

The most direct evidence that a merge added information is that the corpus statistics move. Same store, same queries, before and after:

And a query that could not be posed the day before returns both sites in one result set. Searching apm-demonstration — a mission run entirely on the other site — gives 200 hits spanning 08-14 to 08-17, authored by joe (83), mission-control/sync (44), codex-4 (39), claude-2 (24), claude-7 (9). None of that existed in this store yesterday.

Document frequency also turns out to be a usable salience signal, once you exclude telemetry authors. Out of 147,560 documents: prelim 8,108 and agency 3,137 are ambient and carry little information; wysiwyg 72 and latexml 40 are bounded work episodes you could reconstruct; oomd 2 is a thing that happened once. The low-frequency end is where the forgettable work lives — and the things easiest to forget are precisely the ones nobody is using daily.

6. What we are adopting

• Replay in causal order. Ascending (at, id), not the order the API hands you. Referential integrity will otherwise reject every reply chain, loudly and correctly.
• Never let a scan advance while a bulk insert lands behind it. Do the insertion phase, then reset the frontier, then scan once. Not concurrently.
• Treat a basis written during bulk insertion as void. Clear it and re-earn it. It costs ten minutes and it is the difference between a proof and a coincidence.
• Sort by time, not by relevance, when asking what a body of work was. Relevance ranking surfaced one cluster of a term's history and hid another entirely; chronological ordering exposed both.

The general shape is one we keep meeting: a system where every component behaves correctly, and the composition still asserts something untrue. The basis-tx was not a lie told by a broken scan. It was a true statement about a scan, promoted into a false statement about a store.

7. The part that needs a ruling

The four practices above are discipline: they work, and they depend on whoever runs the next replay remembering them. The open question is whether the store should enforce any of it.

A narrower version: keep the basis, but record that a behind-frontier write occurred, so a consumer can see the claim is provisional. That is cheaper and weaker. Either way the current behaviour — silently keeping a basis that a bulk insert has invalidated — is the one option that should not survive.

Related: this is the same defect family as a write path that validates shape but not identity. Both produce values that round-trip correctly and mean something false, and neither fails loudly.