feat(docker): bake checkpoints from ModelScope at build time

[cgq0816]

Why

A host that cannot reach huggingface.co cannot build the quickstart image either. The checkpoints are also published on the ModelScope mirror, so a build can bake them in and the image then runs with no Hub access at all.

What

docker/prefetch_modelscope.py lists the mirror repository, streams the checkpoint's own files with resume, verifies each file's size and, when the mirror publishes a digest, its SHA-256 against the repository metadata, and writes them into the image's hub cache laid out the way snapshot_download reads offline. Nothing under laya/ changes: Agent, the Router that laya-serve builds, laya.cli and the integrations keep their repo ids and resolve to the baked snapshot.

One build argument selects the checkpoint: MODELSCOPE_MODEL=multilingual by default, or english, typed-decisions, all, or explicit repo[:subfolder] specs. Empty by default, so a plain docker build . behaves exactly as before. compose.modelscope.yaml sets the multilingual type for both services and keeps them on the baked cache with HF_HUB_OFFLINE=1; it repeats the arguments for laya-serve, because an override for laya never reaches it.

Checkpoints of one repository bake into a single snapshot: a cached revision resolves to one directory, so baking the root checkpoint and a subfolder separately would leave only the last refs/main resolvable. A download that fails its size or digest check publishes nothing, and the snapshot is renamed into place only after every file is verified.

The SDK guide link now points to the repository README, fixing a pre-existing broken relative link that blocked strict documentation builds.

Verification

Built the image and ran it with no network: the one-shot quickstart routed English input to english and German input to multilingual, both returning real predictions, and laya-serve answered GET /health and POST /v1/systemone the same way. Also checked the documented Compose combination behind a published port. 20 offline tests in tests/test_modelscope_prefetch.py, 121 packaging checks, and 87 Compose environment checks pass. All four ModelScope Compose combinations parse successfully; strict documentation build, Ruff, and compile checks pass. Rechecked offline English/German inference using the existing baked image with a fresh volume, and confirmed that a populated volume preserves its old ref.

Limits

Only what was baked loads offline. An image that baked only multilingual answers 500 inference failed on English traffic unless the client pins model or the image also bakes english; this is documented in docs/docker.md, together with the recovery knobs.

A Compose deployment must use an empty model-cache volume: select a fresh LAYA_CACHE_VOLUME, or remove the old cache with docker compose down --volumes using the same Compose files and volume setting. Docker does not re-seed a populated volume, so it otherwise hides the baked snapshot. This prerequisite is documented in both compose.modelscope.yaml and docs/docker.md.

When the mirror supplies no digest, verification is size-only and cannot detect a same-size substitution. Mirror metadata is not an independently pinned digest; the selected mirror account is the operator's supply-chain decision.

[Bruce-Yii]

The previous hold on this was wrong: this PR is not unverifiable. About 80% of it verifies without a build, and I have that 80% green. What is actually open is a one-paragraph documentation bug — plus a CI approval only you can do.

Verified without a build

tests/test_modelscope_prefetch.py → 20 passed. tests/test_packaging.py → 121 passed. No new third-party dependency exists to declare: docker/prefetch_modelscope.py is pure stdlib, modelscope is never imported anywhere in the repo, pyproject.toml is untouched, and the script is standalone behind if __name__ == "__main__". Zero files under laya/ change, so the default Hugging Face path is untouched and plain docker build . is a no-op (Dockerfile:36 defaults the arg empty, :74 short-circuits).

The tests are not vacuous, which matters given the claim. The bake tests drive a stubbed ModelScope API and then assert against the real snapshot_download(..., local_files_only=True) plus real file bytes, so a regressed cache layout cannot pass; the three negative tests assert a failed download leaves nothing behind. Your checkpoint-resolution logic, the digest/size checks and the fail-closed direction are all genuinely covered.

The only part that still needs a build is the Docker/Compose layer — 196 of 962 lines. So this splits cleanly by file: prefetch_modelscope.py + its test + the one ci.yml line (766 lines, green today) can land on its own merits, leaving the image wiring for a build-capable review.

The actual bug: the baked weights are shadowed by the cache volume

This is why nobody can reproduce your end-to-end claim on an existing host, and it is a recipe problem, not a code problem.

• Dockerfile:56 sets HF_HOME=/home/laya/.cache/huggingface, and :78 bakes into --cache-dir "$HF_HOME/hub".
• compose.yaml:29 mounts model-cache:/home/laya/.cache/huggingface — the exact parent directory of what you bake into.
• compose.yaml:34 gives that volume a stable, persistent name: ${LAYA_CACHE_VOLUME:-${COMPOSE_PROJECT_NAME}_model-cache}.

Docker seeds a named volume from the image only when the volume is empty. On any host that has run the documented quickstart before — which is the default path — that volume already holds a Hub-downloaded snapshot whose refs/main points at a Hub commit. Your build bakes the ModelScope tip into the image; the volume is never re-seeded, so the baked tip is invisible at runtime. The loader resolves refs/main to the old Hub commit and silently serves the old weights, while the operator reasonably believes MODELSCOPE_MODEL=english took effect. And with HF_HUB_OFFLINE defaulting to 1 (compose.modelscope.yaml:32,39), a missing or divergent ref becomes a hard load failure with no network to fall back to.

compose.modelscope.yaml:9 asserts the opposite of what happens on a used host — "the weights land in the hub cache the image already mounts as the model-cache volume" — which is only true for a fresh volume. And the prerequisite is documented nowhere: docs/docker.md mentions LAYA_CACHE_VOLUME only in the environment-variable table, and docker compose down --volumes only under deleting downloaded weights.

Fix: state the prerequisite in compose.modelscope.yaml and in the ModelScope section of docs/docker.md (use a fresh LAYA_CACHE_VOLUME, or down --volumes first), or bake to a path outside the mounted volume. The first is a paragraph.

The thing only you can do: this PR has never run CI

All four check suites on f69bf111 are conclusion=action_required with latest_check_runs_count=0. The cause is that the head repo is a fork (cgq0816/laya), so GitHub is holding the pull_request workflows for approval. So #720 is not red — it is unrun, and there is no signal from it in either direction. Approving that run is a one-click unblock, and the py3.10–py3.13 lane would exercise the new suite, which I have already reproduced green locally.

For the record, neither of the two repo-level causes people tend to blame is implicated: docs (strict build) and dependency CVEs did not run here at all.

Minor

• The body says "19 new offline tests"; there are 20 (your second commit added test_main_drives_the_whole_argument).
• The suite is wired into ci.yml but not release.yml, which is a pre-existing gap — release.yml currently omits 37 of 79 test files — so not something this PR introduced.
• The "verifies each file's SHA-256" claim is conditional: the check only fires when the mirror publishes a digest, and an absent digest degrades to size-only. Worth a line in the docs' limits section, along with the absence of a digest pin, since the mirror account is the operator's supply-chain decision.
• compose.modelscope.yaml is not in tests/test_compose_env.py's COMPOSE_FILES list, so it gets no env-passthrough or device coverage. Parse coverage does exist via the four stacked docker compose config --quiet combinations in docker.yml.

Recommendation: mergeable on the code, not on the recipe. Land Subset A now if you want it reviewed without a build; the volume-shadowing paragraph is the whole of what stands between this and a clean approval.

[cgq0816]

@Bruce-Yii, the review items are addressed on the current head, 8820ba4:

• Cache-volume prerequisite: 8004e75 documents in both compose.modelscope.yaml and the ModelScope section of docs/docker.md that deployment needs an empty volume: use a fresh LAYA_CACHE_VOLUME, or remove the old volume with docker compose down --volumes using the same Compose files and volume setting. A populated volume is not re-seeded and can hide the baked snapshot.
• Digest claims: d717361 clarifies that SHA-256 is checked only when the mirror publishes a digest; otherwise verification is size-only and cannot detect a same-size substitution. Mirror metadata is not an independently pinned digest. The PR description now states these limits too, and the script's example uses the correct --model argument.
• Minor items: the PR body now says 20 offline tests, and 8417cb0 adds compose.modelscope.yaml to the Compose environment suite. The pre-existing release-workflow coverage gap is unchanged.
• Documentation gate: the strict build exposed an existing SDK-guide link pointing outside the documentation directory; 8820ba4 changes it to the repository README link. The strict build now passes with no griffe: diagnostics.

Verification: 20 prefetch tests, 121 packaging checks, and 87 Compose environment checks pass. All four ModelScope Compose combinations parse successfully; Ruff and compilation checks pass. Using the existing baked image, I also verified offline English/German inference with a fresh volume and confirmed that reusing a populated volume preserves its old ref and shadows the image's baked ref. This was runtime verification of the existing image, not a new image build.

CI remains awaiting maintainer approval. All four check suites on 8820ba4 still report action_required with zero check runs. The authenticated fork-owner account has no write permission on the upstream repository, so I cannot approve those workflows. Maintainer approval is the remaining unblock; local results above do not imply CI has run.

[NandhaKishorM]

Merged for 0.3.23.

Both review items are addressed on the head I took: the cache-volume prerequisite is documented in compose.modelscope.yaml and in the ModelScope section of docs/docker.md, and the digest wording now says plainly that SHA-256 is checked only when the mirror publishes one and that verification is size-only otherwise. The second one is the part I cared about, because a size-only check that reads as a digest check is worse than no claim.

To be explicit about scope, since this adds a second place weights can come from: nothing under laya/ changes, inference stays local, and the mirror is a build-time source for the same checkpoints rather than a runtime service. That is the line, and this stays on the right side of it.

tests/test_modelscope_prefetch.py and tests/test_packaging.py pass on the integrated branch; the image build itself I have not run.

Thank you.