Memory-tree corruption reported as "non-fatal" for 34min while the UI showed successful syncs; recovery then wiped the source registry #5820

Summary

A user's memory_tree/chunks.db was corrupt for at least 34 minutes while the UI reported successful syncs. Every tree ingest failed against the malformed database and was logged at WRN as "non-fatal", so nothing surfaced to the user until a different code path treated the same corruption as fatal and quarantined the database — destroying the ingested-source registry and leaving memory sync broken.

Two separate defects are in play. The corruption cause is already fixed. The reporting defect is not, and it is the reason this went unnoticed.

Evidence

1. Silent failure — 747 ingest failures reported as non-fatal

From a live session log, 01:56:52 → 02:30:25 (34 minutes, 747 occurrences):

WRN tinymemory_core::sync::pipelines::host [memory_sync] tree ingest failed;
    skill store remains authoritative
    error=memory-tree ingest failed for source `github:…`: database disk image is malformed
WRN tree_ingest [archivist] tree ingest failed (non-fatal): source_id=conversations:agent

During this window the Sync History UI showed success — Gmail 250 items, GitHub 979 items. The fetch succeeded; the tree ingest failed every time. The user had no way to know their memory tree was receiving nothing.

"skill store remains authoritative" is doing a lot of work here: it turns a corrupt primary store into an apparently-healthy degraded mode, indefinitely, with no escalation.

2. Detection came from an unrelated path, 10 hours later

12:33:04 ERR memory.tree_jobs_worker_corrupt failed: Failed to claim next mem_tree_jobs row:
         database disk image is malformed: Error code 779

tree_jobs appears 0 times in the prior session's log and 1 time in this one. The job-claim path is the first code to treat SQLITE_CORRUPT as fatal; the ingest path had been swallowing the identical error for hours.

3. The quarantine then destroyed working state

chunks.db.corrupt-20260827T070304Z    180,985,856 bytes (172 MB)   quarantined
chunks.db                                 389,120 bytes            fresh, empty

mem_tree_ingested_sources     quarantined: 1209 rows      fresh: 0 rows
mem_tree_chunks               quarantined: 2072 rows      fresh: 0 rows

Quarantine-and-rebuild is the right instinct and it correctly preserved rather than deleted the file. But because the registry lives in the same database, every subsequent sync now fails:

ERR rpc.invoke_method failed: not found: ai.tinyhumans.tinymemory.Error.NotFound:
    no memory source registered as src_365910b2…    (×4 sources)
WRN [memory_sources] apply_all_in_rpc: sync trigger failed for source … (×4)

The user's memory sync is now hard-broken, as a consequence of the recovery, with no prompt, no explanation, and no offer to re-ingest.

4. The damage is localised and likely recoverable

PRAGMA quick_check → Tree 25 page 43608 cell 0:   invalid page number 44182
                     Tree 25 page 43608 cell 355: invalid page number 44178
damaged trees: 18, 25          page_count: 44167       page_size: 4096
63 tables still readable; mem_tree_chunks still returns 2072 rows

Two damaged B-trees out of a 172 MB file. A sqlite3 .recover would very likely retrieve most of it. Nothing in the product attempts this or tells the user the file exists.

Root cause of the corruption itself — already fixed

1bf2037a0 "Stop booting the second in-process memory engine" (2026-08-25), shipped to main via #5725:

"The host booted two memory engines over one SQLite file: the loaded TinyMemory module, reached over the bus, and a second in-process copy of the engine crate. context.rs created both in the same if plan.memory block."

Two independent engines writing one SQLite file. Compounding it, chunks.db runs journal_mode=delete, not WAL — there is a chunks.db-journal and no -wal/-shm, unlike memory/memory.db which does use WAL. Rollback-journal mode has none of WAL's multi-connection safety, so concurrent writers corrupt it readily.

So the cause is closed. But every workspace written by a pre-#5725 build may carry latent damage that will surface exactly like this — at an arbitrary later moment, via whichever code path first treats it as fatal.

What this issue asks for

The corruption cause needs no fix. These do:

1. Corruption must not be WRN / "non-fatal". A malformed primary store is not a degraded mode to run in for 34 minutes. Classify SQLITE_CORRUPT consistently across ingest and job-claim — today the two paths disagree, which is the whole bug.
2. The UI must not report a successful sync when tree ingest failed. Sync History showed 250 and 979 items ingested while every tree write failed. Same family as Coding-session import reports a failure banner for an import that succeeds #5802 (a banner that did not match reality), opposite direction.
3. Quarantine should not silently strand the user. After rebuild, the registry is empty and every sync fails with no memory source registered — a confusing error unrelated to the actual cause. Detect the post-recovery empty state and either re-register sources automatically or tell the user plainly what happened and what to do.
4. Surface the quarantined file. 172 MB of the user's indexed memory is sitting on disk and nothing mentions it. At minimum log the path prominently; ideally attempt .recover and report what was salvaged.
5. Consider a startup integrity check for workspaces written by pre-Route the recall, reset and flush paths through the contract #5725 builds, since latent damage is expected in that population.

Worth deciding separately: whether chunks.db should be on WAL like memory.db is, as defence in depth.

Environment

• main @ 04075d537, dev build, macOS
• Workspace in continuous use since 2026-07-22
• Corruption first visible in logs 2026-08-27 01:56:52; quarantine fired 12:33:04
• The quarantined file is intact and has not been modified — recovery is still possible

Notes

Related: #5802 (failure banner for a successful import) — the mirror image of item 2. #5801/#5808 (capability advertised but unserved) — the same theme of a failure surfacing at runtime through the wrong layer.