Skip to content

chore(memory): bump tinymemory to v1.23.4; derive-free flow digests and keyed memory - #7043

Merged
M3gA-Mind merged 4 commits into
tinyhumansai:mainfrom
M3gA-Mind:feat/tinymemory-automatic-consolidation
Oct 6, 2026
Merged

M3gA-Mind merged 4 commits into
tinyhumansai:mainfrom
M3gA-Mind:feat/tinymemory-automatic-consolidation

Conversation

@M3gA-Mind

@M3gA-Mind M3gA-Mind commented Oct 6, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Problem

Solution

  • vendor/tinymemory → 581a2bc853 (v1.23.4). Both lockfiles (Cargo.lock, crates/openhuman-app/Cargo.lock) change only these lines:

    • the three tinymemory crate versions, 1.23.1 → 1.23.4;
    • their sha2 edge, 0.10.9 → 0.11.0, a version already in the lockfiles.

    Every changed line was checked, and cargo metadata --locked passes for both manifests. They were edited by hand, because cargo update -p tinymemory-* also re-resolved unrelated windows-sys/getrandom edges.

  • memory/brain.rs: ingested.job is an Option; enqueue it only when present.

  • memory/sources/sync.rs: no BuildBeliefs jobs when the bound engine's descriptor is Consolidation::Automatic.

  • memory/engine.rs: the hosted EngineSettings literal takes ..EngineSettings::default() (the new consolidation field; None keeps the engine's own).

  • flows/bus/run_digest.rs and flows/memory_tools.rs (remember_keyed): meta.derive = Some(false).

  • memory/ops.rs merge_meta: it copies an explicit list of host fields, so a host's derive was silently dropped. derive is now on the list. Without it, the keyed-memory change above would have done nothing.

  • docs/specs/memory-v2.md: a long document is several writes. Hosted billing is per write, so a 25 MB file is about 100 billed writes. Engines that build on their own get no build job.

  • Module pins: N/A. tinymemory is no longer a loadable module. 5c22db1 and ce2ab69 removed its registry record and host integration. It appears in none of these, and no workflow hard-codes a tinymemory archive or sha256:

    • scripts/ci/module-provider-pins.json;
    • scripts/ci/module-pin-exemptions.json;
    • scripts/ci/check-module-pins.mjs;
    • crates/openhuman-core/src/modules/registry/records_*.rs.

    The gate that applies, scripts/ci/check-submodule-monotonic.mjs, passes: 18 submodules, none moved backwards.

Submission Checklist

  • Tests added or updated (happy path + at least one failure / edge case):
    • a_sync_queues_a_belief_build_only_for_an_engine_that_waits_for_one: an on-demand engine queues one build, an Automatic engine none. Revert-check: dropping the gate fails it.
    • scrubbing_keeps_the_page_breaks_a_document_is_chunked_by: the scrubber keeps the U+000C page breaks tinymemory chunks on.
    • Digest and keyed-memory assertions that meta.derive == Some(false) round-trips. These also pin the merge_meta fix.
  • Diff coverage ≥ 80%: to be measured by CI on this head (CI Fast, diff-cover).
  • Coverage matrix updated: N/A, a dependency bump plus memory-internal behaviour; no matrix row changes.
  • All affected feature IDs from the matrix are listed under ## Related: N/A, none.
  • No new external network dependencies introduced: none.
  • Manual smoke checklist updated if this touches release-cut surfaces: N/A.
  • Linked issue: N/A, no issue.

Impact

  • Core memory only (desktop and CLI); no UI or RPC change.
  • With CortexDB, events are now written as prose with the envelope in labels (v3). Stores written before keep reading: tinymemory reads both layouts, so no migration is needed.
  • A long document becomes several writes (about 100 for a 25 MB file).
  • Forgetting very large documents on CortexDB 0.10.4 is slow: seconds per ~240 KiB event. This was measured upstream and is tracked separately.

Related


AI Authored PR Metadata (required for Codex/Linear PRs)

Linear Issue

  • Key: N/A
  • URL: N/A

Commit & Branch

  • Branch: feat/tinymemory-automatic-consolidation
  • Commit SHA: f5f5cbfa07

Validation Run

  • pnpm --filter openhuman-app format:check: N/A (no frontend change); cargo fmt --all -- --check and the app manifest's fmt check both pass.
  • pnpm typecheck: N/A, no TypeScript change.
  • Focused tests: cargo test -p openhuman --lib memory:: (165 passed) and flows:: (564 passed).
  • Rust fmt/check:
    • cargo check --workspace and on the app manifest: pass.
    • cargo clippy --keep-going -p openhuman --tests -- -D warnings and on the app manifest: pass.
    • cargo metadata --locked for both manifests: pass.
    • pnpm rust:layout, check-module-pins.mjs and check-submodule-monotonic.mjs: pass.
  • Tauri fmt/check: included above (the app manifest).

Validation Blocked

  • None.

Behavior Changes

  • Intended behavior change:
    • no belief-build jobs from synced sources on engines that build on their own;
    • flow digests and keyed memory are not derived from;
    • a host's derive reaches the stored item.
  • User-visible effect: none directly. Fewer derived "memories" sourced from flow bookkeeping.

Parity Contract

  • Legacy behavior preserved: on-demand engines still get one build job per touched brain source, as before. Existing CortexDB stores read unchanged.
  • Guard/fallback/dispatch parity checks: the scrubber still runs on every write and keeps page breaks (tested).

Duplicate / Superseded PR Handling

  • Duplicate PR(s): none.
  • Canonical PR: this one.
  • Resolution: N/A.

Summary by CodeRabbit

  • Memory
    • Flow-generated summaries and key/value memories are marked so they do not contribute derived beliefs.
    • Automatic consolidation no longer queues a separate belief-build job. Other consolidation modes continue to queue one for touched sources.
  • Bug Fixes
    • Secret scrubbing now preserves page breaks and the text that follows them.
  • Documentation
    • Clarified that large documents are split into smaller writes at page breaks and headings. Reads still show one item, while search results identify the matching page or section.
    • Documented large-document handling and consolidation behavior.

- Brain ingest enqueues the build job only when tinymemory hands one back
  (`Ingested::job` is now an Option: none for an engine that rebuilds
  beliefs on its own).
- Synced sources queue no belief build for such an engine either.
- The hosted engine settings take the new `consolidation` default.
- A test pins that scrubbing keeps the U+000C page breaks a document is
  chunked by.
- The memory spec notes that a long document is several billed writes.
…-free

- vendor/tinymemory 8d9a24f7 -> 581a2bc8 (v1.23.4: tinyhumansai#203-tinyhumansai#211). Cargo.lock
  and crates/openhuman-app/Cargo.lock change only the three tinymemory
  versions and their sha2 0.10.9 -> 0.11.0 edge.
- Flow run digests and flow keyed memory are stored with derive: false:
  searchable, but nothing is derived from them as memory about the user.
- merge_meta carries the host's derive onto the stored item; it copies an
  explicit field list, so the flag was otherwise dropped.
@tinysweeper

tinysweeper Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

Tiny Sweeper review

Tiny Sweeper reviewed this change across 6 lane(s) and found 2 active actionable finding(s). Detailed lane evidence and any incomplete work are listed below.

State: Reviewing pending checks
Priority: medium
Reviewed head: 3b10ab94c0fa
Updated: 1791323359 (Unix time)

Review snapshot

Change surface Files Review signal Count
Production 6 Active findings 3
Tests 5 Noted findings 0
Documentation 1 Resolved findings 5
Configuration 0 Pending checks/questions 4

Completeness: Complete
Test assessment: No supported feature-to-test mapping was available; this does not mean tests are absent or passed.

What changed

The review could not produce a supported behavioral summary; inspect the cited changed surface and lane details below.

Features

None identified with supported citations.

Tests

No supported feature-to-test mapping was produced. Test execution is not inferred.

Findings

  • medium · critique · Verify the dependency implements documented document chunking — This test exercises consolidation queueing, but it still uses `ReferenceEngine` without verifying the document-chunking behavior that the ingest path relies on. A passing test ther (crates/openhuman\-core/src/memory/brain\_tests\.rs:220)
  • medium · security · Verify the dependency implements documented document chunking — This test checks only the number of queued jobs after ingestion; it never inspects the stored document or its chunks. Consequently, it cannot detect a dependency regression where i (crates/openhuman\-core/src/memory/brain\_tests\.rs:233)
  • medium · e2e · No end-to-end test drives ingest or sync job queueing — The new unit tests (brain_tests.rs `an_ingest_queues_a_belief_build_only_for_an_engine_that_waits_for_one`, sync_tests.rs `a_sync_queues_a_belief_build_only_for_an_engine_that_wait (crates/openhuman\-core/src/memory/sources/sync\.rs:137)

Resolved this pass

  • Pin the no-job path of ingest for automatic engines
  • Pin the no-job path of ingest for automatic engines
  • Pin the no-job path of ingest for automatic engines
  • Pin the no-job path of ingest for automatic engines
  • Verify the dependency implements documented document chunking

Pending checks: Rust E2E (mock backend), Build Playwright E2E Artifact, E2E (Playwright / web lane), Desktop E2E (full suite, 3 OS)

Before merge

  • Wait for Rust E2E (mock backend), Build Playwright E2E Artifact, E2E (Playwright / web lane), Desktop E2E (full suite, 3 OS).

How this fits together

flowchart LR
  n0["...writes_run_digest_entry_for_completed_run<br/>changed"]:::changed
  n1["flow_namespace"]:::impacted
  n2["digest_ignores_cancelled_run"]:::impacted
  n3["digest_ignores_failed_run"]:::impacted
  n4["...treats_completed_with_warnings_as_success"]:::impacted
  n5["upsert_flow"]:::impacted
  n0 -->|calls| n1
  n0 -->|tests| n1
  n0 -->|calls| n5
  n0 -->|tests| n5
  n2 -->|calls| n1
  n2 -->|tests| n1
  n2 -->|calls| n5
  n2 -->|tests| n5
  n3 -->|calls| n1
  n3 -->|tests| n1
  n3 -->|calls| n5
  n3 -->|tests| n5
  n4 -->|calls| n1
  n4 -->|tests| n1
  n4 -->|calls| n5
  n4 -->|tests| n5
  classDef changed fill:#0d4429,stroke:#238636,color:#e6edf3
  classDef impacted fill:#161b22,stroke:#6e7681,color:#c9d1d9
  classDef flagged fill:#5a1e02,stroke:#d93f0b,color:#ffffff
  classDef blocking fill:#67060c,stroke:#f85149,color:#ffffff
Loading
Agent review details

critique

  • Conclusion: Success
  • Scope reviewed: all assigned evidence
  • Lane summary: The change adds focused coverage for consolidation-dependent ingest queueing, and the earlier no-job-path concern is resolved. The dependency chunking concern remains unresolved, so this revision is not ready to merge. _Code retrieval was unavailable (model: ladder embeddings returned 400 Bad Request: {"error":{"message":"unknown ladder vectors; known ladders are flash (also chat-v1, flash-v1), instant (also no-think, instant-v1), reasoning (also deepseek), max-reasoning (also max-reasoning-v1), deepseek-flash (also reasoning-v1, agentic-v1), deep (also luna), scribe, uncensored, vectors-oai3 (also embeddings-oai3-v1), vision (also vision-v1, multimodal-v1), image (also images-v1, image-v1), vi), so this review saw the diff alone._ _Memory was unavailable (model: cortex: v1/recall: error sending request for url (http://cortexdb:3141/v1/recall\)\), so this review ran without it._
  • Evidence: crates/openhuman\-core/src/memory/brain\_tests\.rs — Verify the dependency implements documented document chunking

security

  • Conclusion: Success
  • Scope reviewed: all assigned evidence
  • Lane summary: The new test fixes the earlier automatic-engine queue concern by pinning both consolidation modes. The documented document-chunking behavior remains unverified. _Code retrieval was unavailable (model: ladder embeddings returned 400 Bad Request: {"error":{"message":"unknown ladder vectors; known ladders are flash (also chat-v1, flash-v1), instant (also no-think, instant-v1), reasoning (also deepseek), max-reasoning (also max-reasoning-v1), deepseek-flash (also reasoning-v1, agentic-v1), deep (also luna), scribe, uncensored, vectors-oai3 (also embeddings-oai3-v1), vision (also vision-v1, multimodal-v1), image (also images-v1, image-v1), vi), so this review saw the diff alone._ _Memory was unavailable (model: cortex: v1/recall: error sending request for url (http://cortexdb:3141/v1/recall\)\), so this review ran without it._
  • Evidence: crates/openhuman\-core/src/memory/brain\_tests\.rs — Verify the dependency implements documented document chunking

tests

  • Conclusion: Success
  • Scope reviewed: all assigned evidence
  • Lane summary: The new test for `Brain::ingest` properly pins the no-build-job behaviour for `Consolidation::Automatic` engines, mirroring the sync-side test, so the earlier finding about the ingest path is resolved; the one on the dependency's chunking stays open. No new issues in the incremental diff. (1 finding discarded for not matching a changed line) (1 earlier finding(s) still open) _Code retrieval was unavailable (model: ladder embeddings returned 400 Bad Request: {"error":{"message":"unknown ladder vectors; known ladders are flash (also chat-v1, flash-v1), instant (also no-think, instant-v1), reasoning (also deepseek), max-reasoning (also max-reasoning-v1), deepseek-flash (also reasoning-v1, agentic-v1), deep (also luna), scribe, uncensored, vectors-oai3 (also embeddings-oai3-v1), vision (also vision-v1, multimodal-v1), image (also images-v1, image-v1), vi), so this review saw the diff alone._ _Memory was unavailable (model: cortex: v1/recall: error sending request for url (http://cortexdb:3141/v1/recall\)\), so this review ran without it._

commits

  • Conclusion: Neutral
  • Scope reviewed: all assigned evidence
  • Lane summary: Nothing sensitive found in what this pull request commits.

description

  • Conclusion: Success
  • Scope reviewed: all assigned evidence
  • Lane summary: This revision closes the `merge_meta` gap that made the `derive` flag silently disappear on host-written items, sets `derive: Some(false)` on flow digests and keyed memory with round-trip tests, and pins both the sync and ingest no-job paths with tests. The PR description matches what the diff does; the two findings from the last pass are addressed and nothing new blocks the merge. _Code retrieval was unavailable (model: ladder embeddings returned 400 Bad Request: {"error":{"message":"unknown ladder vectors; known ladders are flash (also chat-v1, flash-v1), instant (also no-think, instant-v1), reasoning (also deepseek), max-reasoning (also max-reasoning-v1), deepseek-flash (also reasoning-v1, agentic-v1), deep (also luna), scribe, uncensored, vectors-oai3 (also embeddings-oai3-v1), vision (also vision-v1, multimodal-v1), image (also images-v1, image-v1), vi), so this review saw the diff alone._ _Memory was unavailable (model: cortex: v1/recall: error sending request for url (http://cortexdb:3141/v1/recall\)\), so this review ran without it._

e2e

  • Conclusion: Neutral
  • Scope reviewed: all assigned evidence
  • Lane summary: This revision adds a unit test pinning the no-belief-build path for automatic-consolidation engines (previously raised) and a scrubber test for form-feed page breaks, but no end-to-end test drives either the ingest/sync job behaviour or the documented large-document chunking, so both earlier findings still stand at their prior severity. The new derive=false flags are unit-tested only; that is out of my lane. No new e2e gap beyond the two already on record — the lexical candidates in auth-access-control.spec.ts are unrelated console.log text matches, not coverage. (1 finding discarded for not matching a changed line) Waiting on end-to-end jobs: `Rust E2E (mock backend)`, `Build Playwright E2E Artifact`, `E2E (Playwright / web lane)`, `Desktop E2E (full suite, 3 OS)`. (2 earlier finding(s) still open)
  • Unresolved questions/checks: Rust E2E (mock backend), Build Playwright E2E Artifact, E2E (Playwright / web lane), Desktop E2E (full suite, 3 OS)
  • Evidence: crates/openhuman\-core/src/memory/sources/sync\.rs — No end-to-end test drives ingest or sync job queueing
Evidence and run details
  • Models: gpt-5.6-luna, glm-5.3-flash
  • Spend: $0.003064
  • Tokens: 73785 input · 7724 output · 1787 cached · 0 embedding
Head State Pass summary
f5f5cbfa0725 pending 2 active finding(s), 0 resolved finding(s) (at 1791322508)
3b10ab94c0fa pending 3 active finding(s), 5 resolved finding(s) (at 1791323359)

tinysweeper 0.1.0

@coderabbitai

coderabbitai Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 9fb33296-5001-49fa-bcb8-4eb9d77440c3
📥 Commits

Reviewing files that changed from the base of the PR and between f5f5cbf and 3b10ab9.

📒 Files selected for processing (1)
  • crates/openhuman-core/src/memory/brain_tests.rs

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 8 remain after this review.


📝 Walkthrough

Walkthrough

Memory changes update derive metadata on selected stored items, adjust belief-build job queuing for ingestion and automatic consolidation, and update documentation for document chunking.

Changes

Memory updates

Layer / File(s) Summary
Derived-memory metadata
crates/openhuman-core/src/flows/bus/run_digest.rs, crates/openhuman-core/src/flows/bus_subscriber_tests.rs, crates/openhuman-core/src/flows/memory_tools.rs, crates/openhuman-core/src/flows/memory_tools_tests.rs, crates/openhuman-core/src/memory/ops.rs
Run digests and keyed flow memories set meta.derive to false. Tests assert the stored value. merge_meta copies the host’s derive value when present.
Ingestion handling and documentation
crates/openhuman-core/src/memory/brain.rs, crates/openhuman-core/src/memory/engine.rs, crates/openhuman-core/src/memory/guard_tests.rs, docs/specs/memory-v2.md, vendor/tinymemory
Ingestion enqueues zero or one job. Engine settings use defaults for unspecified fields. A test checks that scrubbing preserves page breaks. The documentation describes chunking, reads, search results, and hosted-engine writes. The tinymemory submodule reference changes.
Consolidation job scheduling
crates/openhuman-core/src/memory/sources/sync.rs, crates/openhuman-core/src/memory/sources/sync_tests.rs, crates/openhuman-core/src/memory/brain_tests.rs
Automatic consolidation skips belief-build jobs for touched sources. Tests check that OnDemand queues one job and Automatic queues none; both store one item.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Other

Suggested reviewers: senamakel

Merge Risk: ⚪ Minimal · up to 3b10a

No additional issue is identified in this change. It is mergeable after normal checks.

Architecture Summary

Architecture risk: 🔵 Low · up to 3b10a

The change affects 2 systems.

Changed systems: crates, docs

Architecture concerns
No architecture-level concerns identified.

Review details

Systems and components

  • observed — crates (service) was modified; 11 changed files map to changed impact.
  • observed — docs (service) was modified; 1 changed file maps to changed impact.

Before / after behavior

  • observed — Modified behavior in crates/openhuman-core/src/flows/bus/run_digest.rs: Added comments describing the digest’s intended treatment and set meta.derive to false before storing it.
  • observed — Modified behavior in crates/openhuman-core/src/flows/bus_subscriber_tests.rs: The completed-run digest test adds an assertion that the stored digest has meta.derive == Some(false).
  • observed — Modified behavior in crates/openhuman-core/src/flows/memory_tools.rs: remember_keyed now sets derive to false on the flow metadata before learning the keyed item; previously it passed the metadata directly without this setting.
  • observed — Modified behavior in crates/openhuman-core/src/flows/memory_tools.rs: The learn call now receives the prepared metadata, including derive = false, instead of constructing and passing flow_meta inline.
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the tinymemory update and the derive-free flow digest and keyed-memory changes.
Docstring Coverage ✅ Passed Docstring coverage is 85.71% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 14 functions across 11 files.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

A rabbit checks the memory store,
And marks what need not yield more.
Page breaks stay, and chunks align,
While queued jobs follow mode and sign.
The rabbit hops through tests anew,
Then rests beside a document queue.

Comment @coderabbitai help to get the list of available commands.

@tinysweeper tinysweeper Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tinysweeper found nothing blocking. Approving.

             $0.0122 · 259,792 in / 14,581 out · 19,416 cached (7%)  · gpt-5.6-luna, glm-5.3-flash
critique:    $0.0065 · 116,513 in / 6,585 out  · 8,320 cached (7%)   · gpt-5.6-luna, glm-5.3-flash
security:    $0.0056 · 108,793 in / 3,462 out  · 11,096 cached (10%) · gpt-5.6-luna
tests:       $0.0000 · 14,057 in  / 1,941 out  · 0 cached (0%)       · glm-5.3-flash
description: $0.0000 · 7,668 in   / 154 out    · 0 cached (0%)       · glm-5.3-flash
e2e:         $0.0000 · 7,551 in   / 818 out    · 0 cached (0%)       · glm-5.3-flash

Comment thread crates/openhuman-core/src/memory/brain.rs
@tinysweeper tinysweeper Bot added the priority: p2 Soon. Real but survivable — a rough edge, a gap, a thing that will bite later. label Oct 6, 2026

@tinysweeper tinysweeper Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tinysweeper found nothing blocking. Approving.

             $0.0031 · 73,785 in / 7,724 out · 1,787 cached (2%) · gpt-5.6-luna, glm-5.3-flash
critique:    $0.0008 · 11,082 in / 1,608 out · 0 cached (0%)     · gpt-5.6-luna, glm-5.3-flash
security:    $0.0021 · 32,349 in / 2,405 out · 1,787 cached (6%) · gpt-5.6-luna
tests:       $0.0001 · 7,349 in  / 326 out   · 0 cached (0%)     · glm-5.3-flash
description: $0.0001 · 8,603 in  / 111 out   · 0 cached (0%)     · glm-5.3-flash
e2e:         $0.0000 · 8,351 in  / 1,099 out · 0 cached (0%)     · glm-5.3-flash

Comment thread crates/openhuman-core/src/memory/brain_tests.rs
Comment thread crates/openhuman-core/src/memory/brain_tests.rs
Comment thread crates/openhuman-core/src/memory/sources/sync.rs
@M3gA-Mind
M3gA-Mind merged commit 7578346 into tinyhumansai:main Oct 6, 2026
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

priority: p2 Soon. Real but survivable — a rough edge, a gap, a thing that will bite later.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant