Repository navigation
feat: hydrate islands from JSON payloads - #9987
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
There was a problem hiding this comment.
1 issue found and verified against the latest diff
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="marimo/_islands/_island_generator.py">
<violation number="1" location="marimo/_islands/_island_generator.py:569">
P2: Default-on JSON payload duplicates island data already present in DOM, causing page-size and performance regressions for large outputs.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
There was a problem hiding this comment.
1 issue found across 1 file (changes from recent commits).
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
There was a problem hiding this comment.
Pull request overview
Adds an optional JSON “islands payload” export path so the frontend can hydrate island apps from structured data (cells/code/output metadata) instead of reconstructing everything from DOM attributes and encoded code blocks. This reduces the exporter/runtime contract surface for static HTML generators while preserving DOM parsing as a fallback.
Changes:
- Python:
MarimoIslandGeneratorcan emit an inert<script type="application/vnd.marimo.islands+json">…</script>payload (opt-in viainclude_payload) with per-cell code/output metadata and HTML-sensitive escaping. - Frontend: island parsing can consume schema v1 payloads to build “payload-backed” apps (with DOM anchors still used for visible slots), falling back to DOM parsing when payloads aren’t present/valid.
- Tests/docs: adds payload-focused tests, an HTML snapshot containing the payload, and documentation describing the payload contract.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
marimo/_islands/_island_generator.py |
Adds payload schema/constants and payload/script rendering with JSON escaping; threads include_payload through HTML/body renderers. |
tests/_islands/test_island_generator.py |
Adds unit tests for payload generation, option behavior, and script escaping. |
tests/_islands/snapshots/html-payload.txt |
Snapshot of HTML output including the payload script tag. |
frontend/src/core/islands/constants.ts |
Defines the payload script MIME type constant. |
frontend/src/core/islands/parse.ts |
Adds payload parsing/validation + payload-backed hydration path and disabled-cell support in synthesized marimo files. |
frontend/src/core/islands/bridge.ts |
Ensures payload-backed apps don’t get overridden by full-notebook export context. |
frontend/src/core/islands/__tests__/test-utils.tsx |
Extends test helpers to support data-cell-id. |
frontend/src/core/islands/__tests__/parse.test.ts |
Adds extensive tests for payload parsing/materialization and disabled cell synthesis. |
frontend/src/core/islands/__tests__/bridge.test.ts |
Tests that export-context notebook code is ignored for payload-backed apps. |
docs/guides/exporting/webassembly_html.md |
Documents island payloads, escaping behavior, and post-processing guidance. |
a448b82 to
00020df
Compare
|
rebased this to resolve the merge conflict but now CI seems to fail due to fcf95a2 (?) |
|
@peter-gy maybe there is a cycle or something off about the imports |
13bd507 to
5f21379
Compare
|
Had to bump ruff to Looks like this CI failure did not come up in other commits as other gh flows did not reach the branch where |
|
🚀 Development release published. You may be able to view the changes at https://marimo.app?v=0.23.12-dev11 |
Replaces the AST-injected micropip cell with an islands JSON payload (<script type="application/vnd.marimo.islands+json">, marimo >= 0.24, marimo-team/marimo#9987). build_bootstrap_payload() starts from marimo's own render_payload() and (1) prepends an anchor-less bootstrap cell that `await micropip.install([...])`s the derived dependency list and defines a sentinel, (2) prefixes every user cell's payload code with a bare reference to that sentinel so marimo's dataflow runs the bootstrap first. Anchored cells' outputHtml is copied from the already anywidget-rewritten body so the runtime's parse-time materialization writes back exactly what the DOM shows. Emitted only when the dependency list is non-empty. The notebook MarimoIslandGenerator executes is no longer rewritten for this — the old injector round-tripped every WASM page through ast.unparse (the root of the 0.1.28–0.1.30 title-hoist bugs) and needed a setup-block special case; setup code is an ordinary cell to the islands runtime, so that limitation is gone too. The sentinel must not start with an underscore: marimo makes such names cell-local (NameError in the browser). Why not upstream's include_payload: the islands worker only installs packages listed in a column-zero PEP 723 block of the notebook file it synthesizes from cell bodies, and never carries that block over, so the list is always empty for islands. Retiring this bootstrap once marimo carries dependencies in the payload is tracked in #77. Verified on marimo 0.24.1 / @marimo-team/islands@0.24.1: a notebook importing cowsay (pure Python, not in Pyodide) raised ModuleNotFoundError before and renders after; moving a slider recomputes the dependent cell. Bumps _RENDER_OUTPUT_VERSION to "4" and the marimo floor to >=0.24. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LUPL9tMLYpVmkRCfBuw1M9
Replaces the AST-injected micropip cell with an islands JSON payload (<script type="application/vnd.marimo.islands+json">, marimo >= 0.24, marimo-team/marimo#9987). build_bootstrap_payload() starts from marimo's own render_payload() and (1) prepends an anchor-less bootstrap cell that `await micropip.install([...])`s the derived dependency list and defines a sentinel, (2) prefixes every user cell's payload code with a bare reference to that sentinel so marimo's dataflow runs the bootstrap first. Anchored cells' outputHtml is copied from the already anywidget-rewritten body so the runtime's parse-time materialization writes back exactly what the DOM shows. Emitted only when the dependency list is non-empty. The notebook MarimoIslandGenerator executes is no longer rewritten for this — the old injector round-tripped every WASM page through ast.unparse (the root of the 0.1.28–0.1.30 title-hoist bugs) and needed a setup-block special case; setup code is an ordinary cell to the islands runtime, so that limitation is gone too. The sentinel must not start with an underscore: marimo makes such names cell-local (NameError in the browser). Why not upstream's include_payload: the islands worker only installs packages listed in a column-zero PEP 723 block of the notebook file it synthesizes from cell bodies, and never carries that block over, so the list is always empty for islands. Retiring this bootstrap once marimo carries dependencies in the payload is tracked in #77. Verified on marimo 0.24.1 / @marimo-team/islands@0.24.1: a notebook importing cowsay (pure Python, not in Pyodide) raised ModuleNotFoundError before and renders after; moving a slider recomputes the dependent cell. Bumps _RENDER_OUTPUT_VERSION to "4" and the marimo floor to >=0.24. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LUPL9tMLYpVmkRCfBuw1M9
Replaces the AST-injected micropip cell with an islands JSON payload (<script type="application/vnd.marimo.islands+json">, marimo >= 0.24, marimo-team/marimo#9987). build_bootstrap_payload() starts from marimo's own render_payload() and (1) prepends an anchor-less bootstrap cell that `await micropip.install([...])`s the derived dependency list and defines a sentinel, (2) prefixes every user cell's payload code with a bare reference to that sentinel so marimo's dataflow runs the bootstrap first. Anchored cells' outputHtml is copied from the already anywidget-rewritten body so the runtime's parse-time materialization writes back exactly what the DOM shows. Emitted only when the dependency list is non-empty. The notebook MarimoIslandGenerator executes is no longer rewritten for this — the old injector round-tripped every WASM page through ast.unparse (the root of the 0.1.28–0.1.30 title-hoist bugs) and needed a setup-block special case; setup code is an ordinary cell to the islands runtime, so that limitation is gone too. The sentinel must not start with an underscore: marimo makes such names cell-local (NameError in the browser). Why not upstream's include_payload: the islands worker only installs packages listed in a column-zero PEP 723 block of the notebook file it synthesizes from cell bodies, and never carries that block over, so the list is always empty for islands. Retiring this bootstrap once marimo carries dependencies in the payload is tracked in #77. Verified on marimo 0.24.1 / @marimo-team/islands@0.24.1: a notebook importing cowsay (pure Python, not in Pyodide) raised ModuleNotFoundError before and renders after; moving a slider recomputes the dependent cell. Bumps _RENDER_OUTPUT_VERSION to "4" and the marimo floor to >=0.24. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LUPL9tMLYpVmkRCfBuw1M9
Replaces the AST-injected micropip cell with an islands JSON payload (<script type="application/vnd.marimo.islands+json">, marimo >= 0.24, marimo-team/marimo#9987). build_bootstrap_payload() starts from marimo's own render_payload() and (1) prepends an anchor-less bootstrap cell that `await micropip.install([...])`s the derived dependency list and defines a sentinel, (2) prefixes every user cell's payload code with a bare reference to that sentinel so marimo's dataflow runs the bootstrap first. Anchored cells' outputHtml is copied from the already anywidget-rewritten body so the runtime's parse-time materialization writes back exactly what the DOM shows. Emitted only when the dependency list is non-empty. The notebook MarimoIslandGenerator executes is no longer rewritten for this — the old injector round-tripped every WASM page through ast.unparse (the root of the 0.1.28–0.1.30 title-hoist bugs) and needed a setup-block special case; setup code is an ordinary cell to the islands runtime, so that limitation is gone too. The sentinel must not start with an underscore: marimo makes such names cell-local (NameError in the browser). Why not upstream's include_payload: the islands worker only installs packages listed in a column-zero PEP 723 block of the notebook file it synthesizes from cell bodies, and never carries that block over, so the list is always empty for islands. Retiring this bootstrap once marimo carries dependencies in the payload is tracked in #77. Verified on marimo 0.24.1 / @marimo-team/islands@0.24.1: a notebook importing cowsay (pure Python, not in Pyodide) raised ModuleNotFoundError before and renders after; moving a slider recomputes the dependent cell. Bumps _RENDER_OUTPUT_VERSION to "4" and the marimo floor to >=0.24. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LUPL9tMLYpVmkRCfBuw1M9
Replaces the AST-injected micropip cell with an islands JSON payload (<script type="application/vnd.marimo.islands+json">, marimo >= 0.24, marimo-team/marimo#9987). build_bootstrap_payload() starts from marimo's own render_payload() and (1) prepends an anchor-less bootstrap cell that `await micropip.install([...])`s the derived dependency list and defines a sentinel, (2) prefixes every user cell's payload code with a bare reference to that sentinel so marimo's dataflow runs the bootstrap first. Anchored cells' outputHtml is copied from the already anywidget-rewritten body so the runtime's parse-time materialization writes back exactly what the DOM shows. Emitted only when the dependency list is non-empty. The notebook MarimoIslandGenerator executes is no longer rewritten for this — the old injector round-tripped every WASM page through ast.unparse (the root of the 0.1.28–0.1.30 title-hoist bugs) and needed a setup-block special case; setup code is an ordinary cell to the islands runtime, so that limitation is gone too. The sentinel must not start with an underscore: marimo makes such names cell-local (NameError in the browser). Why not upstream's include_payload: the islands worker only installs packages listed in a column-zero PEP 723 block of the notebook file it synthesizes from cell bodies, and never carries that block over, so the list is always empty for islands. Retiring this bootstrap once marimo carries dependencies in the payload is tracked in #77. Verified on marimo 0.24.1 / @marimo-team/islands@0.24.1: a notebook importing cowsay (pure Python, not in Pyodide) raised ModuleNotFoundError before and renders after; moving a slider recomputes the dependent cell. Bumps _RENDER_OUTPUT_VERSION to "4" and the marimo floor to >=0.24. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LUPL9tMLYpVmkRCfBuw1M9
Replaces the AST-injected micropip cell with an islands JSON payload (<script type="application/vnd.marimo.islands+json">, marimo >= 0.24, marimo-team/marimo#9987). build_bootstrap_payload() starts from marimo's own render_payload() and (1) prepends an anchor-less bootstrap cell that `await micropip.install([...])`s the derived dependency list and defines a sentinel, (2) prefixes every user cell's payload code with a bare reference to that sentinel so marimo's dataflow runs the bootstrap first. Anchored cells' outputHtml is copied from the already anywidget-rewritten body so the runtime's parse-time materialization writes back exactly what the DOM shows. Emitted only when the dependency list is non-empty. The notebook MarimoIslandGenerator executes is no longer rewritten for this — the old injector round-tripped every WASM page through ast.unparse (the root of the 0.1.28–0.1.30 title-hoist bugs) and needed a setup-block special case; setup code is an ordinary cell to the islands runtime, so that limitation is gone too. The sentinel must not start with an underscore: marimo makes such names cell-local (NameError in the browser). Why not upstream's include_payload: the islands worker only installs packages listed in a column-zero PEP 723 block of the notebook file it synthesizes from cell bodies, and never carries that block over, so the list is always empty for islands. Retiring this bootstrap once marimo carries dependencies in the payload is tracked in #77. Verified on marimo 0.24.1 / @marimo-team/islands@0.24.1: a notebook importing cowsay (pure Python, not in Pyodide) raised ModuleNotFoundError before and renders after; moving a slider recomputes the dependent cell. Bumps _RENDER_OUTPUT_VERSION to "4" and the marimo floor to >=0.24. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LUPL9tMLYpVmkRCfBuw1M9
Replaces the AST-injected micropip cell with an islands JSON payload (<script type="application/vnd.marimo.islands+json">, marimo >= 0.24, marimo-team/marimo#9987). build_bootstrap_payload() starts from marimo's own render_payload() and (1) prepends an anchor-less bootstrap cell that `await micropip.install([...])`s the derived dependency list and defines a sentinel, (2) prefixes every user cell's payload code with a bare reference to that sentinel so marimo's dataflow runs the bootstrap first. Anchored cells' outputHtml is copied from the already anywidget-rewritten body so the runtime's parse-time materialization writes back exactly what the DOM shows. Emitted only when the dependency list is non-empty. The notebook MarimoIslandGenerator executes is no longer rewritten for this — the old injector round-tripped every WASM page through ast.unparse (the root of the 0.1.28–0.1.30 title-hoist bugs) and needed a setup-block special case; setup code is an ordinary cell to the islands runtime, so that limitation is gone too. The sentinel must not start with an underscore: marimo makes such names cell-local (NameError in the browser). Why not upstream's include_payload: the islands worker only installs packages listed in a column-zero PEP 723 block of the notebook file it synthesizes from cell bodies, and never carries that block over, so the list is always empty for islands. Retiring this bootstrap once marimo carries dependencies in the payload is tracked in #77. Verified on marimo 0.24.1 / @marimo-team/islands@0.24.1: a notebook importing cowsay (pure Python, not in Pyodide) raised ModuleNotFoundError before and renders after; moving a slider recomputes the dependent cell. Bumps _RENDER_OUTPUT_VERSION to "4" and the marimo floor to >=0.24. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LUPL9tMLYpVmkRCfBuw1M9
Replaces the AST-injected micropip cell with an islands JSON payload (<script type="application/vnd.marimo.islands+json">, marimo >= 0.24, marimo-team/marimo#9987). build_bootstrap_payload() starts from marimo's own render_payload() and (1) prepends an anchor-less bootstrap cell that `await micropip.install([...])`s the derived dependency list and defines a sentinel, (2) prefixes every user cell's payload code with a bare reference to that sentinel so marimo's dataflow runs the bootstrap first. Anchored cells' outputHtml is copied from the already anywidget-rewritten body so the runtime's parse-time materialization writes back exactly what the DOM shows. Emitted only when the dependency list is non-empty. The notebook MarimoIslandGenerator executes is no longer rewritten for this — the old injector round-tripped every WASM page through ast.unparse (the root of the 0.1.28–0.1.30 title-hoist bugs) and needed a setup-block special case; setup code is an ordinary cell to the islands runtime, so that limitation is gone too. The sentinel must not start with an underscore: marimo makes such names cell-local (NameError in the browser). Why not upstream's include_payload: the islands worker only installs packages listed in a column-zero PEP 723 block of the notebook file it synthesizes from cell bodies, and never carries that block over, so the list is always empty for islands. Retiring this bootstrap once marimo carries dependencies in the payload is tracked in #77. Verified on marimo 0.24.1 / @marimo-team/islands@0.24.1: a notebook importing cowsay (pure Python, not in Pyodide) raised ModuleNotFoundError before and renders after; moving a slider recomputes the dependent cell. Bumps _RENDER_OUTPUT_VERSION to "4" and the marimo floor to >=0.24. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LUPL9tMLYpVmkRCfBuw1M9
…oad (#78) * feat(wasm): ship the micropip bootstrap in marimo's islands JSON payload Replaces the AST-injected micropip cell with an islands JSON payload (<script type="application/vnd.marimo.islands+json">, marimo >= 0.24, marimo-team/marimo#9987). build_bootstrap_payload() starts from marimo's own render_payload() and (1) prepends an anchor-less bootstrap cell that `await micropip.install([...])`s the derived dependency list and defines a sentinel, (2) prefixes every user cell's payload code with a bare reference to that sentinel so marimo's dataflow runs the bootstrap first. Anchored cells' outputHtml is copied from the already anywidget-rewritten body so the runtime's parse-time materialization writes back exactly what the DOM shows. Emitted only when the dependency list is non-empty. The notebook MarimoIslandGenerator executes is no longer rewritten for this — the old injector round-tripped every WASM page through ast.unparse (the root of the 0.1.28–0.1.30 title-hoist bugs) and needed a setup-block special case; setup code is an ordinary cell to the islands runtime, so that limitation is gone too. The sentinel must not start with an underscore: marimo makes such names cell-local (NameError in the browser). Why not upstream's include_payload: the islands worker only installs packages listed in a column-zero PEP 723 block of the notebook file it synthesizes from cell bodies, and never carries that block over, so the list is always empty for islands. Retiring this bootstrap once marimo carries dependencies in the payload is tracked in #77. Verified on marimo 0.24.1 / @marimo-team/islands@0.24.1: a notebook importing cowsay (pure Python, not in Pyodide) raised ModuleNotFoundError before and renders after; moving a slider recomputes the dependent cell. Bumps _RENDER_OUTPUT_VERSION to "4" and the marimo floor to >=0.24. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LUPL9tMLYpVmkRCfBuw1M9 * fix(wasm): address review findings on the payload bootstrap - marimo_book.js: re-hydrate anywidget/plotly mounts on the runtime's `marimo-island-source-changed` event. The islands runtime materializes the payload only once the Pyodide worker is ready, i.e. after our DOMContentLoaded hydration, and its innerHTML replacement would have left fresh un-hydrated mounts in place of the hydrated ones. - Bootstrap cell always binds the sentinel: catch-all `except Exception` that prints instead of raising, so a bad wheel or network error degrades to per-cell import errors rather than cancelling every cell on the page (the anchor-less cell has nowhere to show a traceback). - Sentinel prefix is `_ = marimo_book_micropip_done` (an assignment, so a comment-only cell can't display it as its last expression); cells that compile to no statements are left alone; idempotency keys on the exact prefix, not a substring, so a cell that merely mentions the sentinel still gets the dataflow edge. - Install list = import-derived deps ∪ the notebook's own PEP 723 block (same merge as the staged manifest) minus Pyodide-bundled packages, resolved through marimo's own lockfile helper (honours MARIMO_PYODIDE_LOCK_FILE, cached under .marimo_book_cache per Pyodide version, unreadable ⇒ no filtering). Handing a host-pinned numpy==X to micropip would otherwise fail, and bundled-only pages no longer ship a payload at all. - Payload script type + escaping come from marimo's own `_schemas.islands` / `_templates.json_script` so an upstream rename fails loudly at the pin bump; the second BeautifulSoup pass is skipped when the anywidget rewrite changed nothing; the two render calls are folded into one; navigation.instant caveat documented at the append site. - CHANGELOG [Unreleased] folded into one coherent story (floor >=0.24, payload bootstrap); tests offline via a fixture pyodide-lock.json. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LUPL9tMLYpVmkRCfBuw1M9 * fix(pep723): let the notebook's own PEP 723 pins win in the WASM install list wasm_install_packages seeded the merge from the import-derived list and then setdefault'ed the notebook's own block, so on a canonical-name collision the unpinned derived entry won — the opposite of write_pep723_block, which the docstring claimed to mirror. A notebook pinning nltools==0.4.0 in its block shipped an unpinned micropip install. Seed from the block first; derived entries only fill in unlisted names. Found by the new automatic PR review. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LUPL9tMLYpVmkRCfBuw1M9 * docs: list every private marimo module behind the pin's upper bound The pin comment and the CLAUDE.md bump checklist said two private modules; this PR uses five (pyodide_constraints in pep723.py, islands schema + json_script in wasm.py). Flagged by the automatic PR review. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LUPL9tMLYpVmkRCfBuw1M9 * fix(cache): include marimo's pinned Pyodide version in the body signature WASM bodies now bake in a micropip install list filtered against the packages bundled with the Pyodide release marimo pins. The body signature only tracked marimo-book's own output-contract version, so a marimo upgrade that moved to a Pyodide dropping a package would replay a cached body that still omits it (ModuleNotFoundError in the browser) until --rebuild. Track PYODIDE_VERSION — it changes rarely, unlike marimo's version, which stays out of the signature on purpose. (The islands CDN URL in the head has the same shape and predates this PR; it is not a correctness issue because old and new bundles both work.) Flagged by the automatic PR review. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LUPL9tMLYpVmkRCfBuw1M9 --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Adds JSON payloads to island exports. The islands runtime can hydrate from a structured payload instead of recovering source and metadata from each island's DOM. This reduces the island-contract logic we need to maintain in static exporters like https://github.com/marimo-team/quarto-marimo and https://github.com/marimo-team/jupyter-book-marimo.
What changed
MarimoIslandGeneratornow emits a JSON script tag on opt-in:code,outputHtml,outputMimetype,reactive,displayCode, anddisplayOutput.{ "cellId": "1faffc38-d97b-492e-9ffc-7c77d08d0d8fMJUe", "code": "mo.md(r\"\"\"\nI am static markdown.\n\"\"\")", "outputHtml": "<marimo-mime-renderer data-mime='"text/markdown"' data-data='"\u003cspan class=\"markdown prose dark:prose-invert contents\"\u003e\u003cspan class=\"paragraph\"\u003eI am static markdown.\u003c/span\u003e\u003c/span\u003e"'></marimo-mime-renderer>", "outputMimetype": "text/markdown", "reactive": true, "displayCode": false, "displayOutput": true }Follow-ups
outputHtmlcan be potentially huge. we could consider emitting a base64-encoded string inside<script type="application/vnd.marimo.islands+jsonb64">if this becomes an issue. Quarto does the same for their ObservableJS support.MarimoIslandGenerator.from_irAPI which would accept aNotebookSerializationobjectDemo