Skip to content

fix(core,model-profiles): add missing ModelProfile fields, warn on schema drift - #36129

Merged
Mason Daugherty (mdrxy) merged 10 commits into
masterfrom
mdrxy/model-profiles/schema
Mar 23, 2026
Merged

fix(core,model-profiles): add missing ModelProfile fields, warn on schema drift#36129
Mason Daugherty (mdrxy) merged 10 commits into
masterfrom
mdrxy/model-profiles/schema

Conversation

@mdrxy

@mdrxy Mason Daugherty (mdrxy) commented Mar 20, 2026

Copy link
Copy Markdown
Member

PR #35788 added 7 new fields to the langchain-profiles CLI output (name, status, release_date, last_updated, open_weights, attachment, temperature) but didn't update ModelProfile in langchain-core. Partner packages like langchain-aws that set extra="forbid" on their Pydantic models hit extra_forbidden validation errors when Pydantic encountered undeclared TypedDict keys at construction time. This adds the missing fields, makes ModelProfile forward-compatible, provides a base-class hook so partners can stop duplicating model-profile validator boilerplate, migrates all in-repo partners to the new hook, and adds runtime + CI-time warnings for schema drift.

Changes

langchain-core

  • Add __pydantic_config__ = ConfigDict(extra="allow") to ModelProfile so unknown profile keys pass Pydantic validation even on models with extra="forbid" — forward-compatibility for when the CLI schema evolves ahead of core
  • Declare the 7 missing fields on ModelProfile: name, status, release_date, last_updated, open_weights (metadata) and attachment, temperature (capabilities)
  • Add _warn_unknown_profile_keys() in model_profile.py — emits a UserWarning when a profile dict contains keys not in ModelProfile, suggesting a core upgrade. Wrapped in a bare except so introspection failures never crash model construction
  • Add BaseChatModel._resolve_model_profile() hook that returns None by default. Partners can override this single method instead of redefining the full _set_model_profile validator — the base validator calls it automatically
  • Add BaseChatModel._check_profile_keys as a separate model_validator that calls _warn_unknown_profile_keys. Uses a distinct method name so partner overrides of _set_model_profile don't inadvertently suppress the check

langchain-profiles CLI

  • Add _warn_undeclared_profile_keys() to the CLI (cli.py), called after merging augmentations in refresh() — warns at profile-generation time (not just runtime) when emitted keys aren't declared in ModelProfile. Gracefully skips if langchain-core isn't installed
  • Add guard test test_model_data_to_profile_keys_subset_of_model_profile in model-profiles — feeds a fully-populated model dict to _model_data_to_profile() and asserts every emitted key exists in ModelProfile.__annotations__. CI fails before any release if someone adds a CLI field without updating the TypedDict

Partner packages

  • Migrate all 10 in-repo partners to the _resolve_model_profile() hook, replacing duplicated @model_validator / _set_model_profile overrides: anthropic, deepseek, fireworks, groq, huggingface, mistralai, openai (base + azure), openrouter, perplexity, xai
  • Anthropic retains custom logic (context-1m beta → max_input_tokens override); all others reduce to a one-liner
  • Add pr_lint.yml scope for the new model-profiles package

@github-actions github-actions Bot added core `langchain-core` package issues & PRs fix For PRs that implement a fix internal model-profiles `langchain-model-profiles` package issues & PRs size: M 200-499 LOC labels Mar 20, 2026
@codspeed-hq

codspeed-hq Bot commented Mar 20, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 65.01%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 2 improved benchmarks
❌ 4 regressed benchmarks
✅ 19 untouched benchmarks
⏩ 15 skipped benchmarks1

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
WallTime test_async_callbacks_in_sync 20.5 ms 15.1 ms +35.57%
WallTime test_init_time 2.8 ms 3.2 ms -12.41%
WallTime test_init_time 2.8 ms 3.2 ms -11.28%
WallTime test_init_time 140.8 ms 124.8 ms +12.85%
Simulation test_init_time 652.1 µs 1,863.5 µs -65.01%
Simulation test_init_time_with_client 1.8 ms 3 ms -39.87%

Comparing mdrxy/model-profiles/schema (337e788) with master (5ffece5)

Open in CodSpeed

Footnotes

  1. 15 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

Wrap `self.profile` access in `_set_model_profile` and
`_check_profile_keys` with try/except AttributeError so subclasses
that override `__getattribute__` to raise on "profile" don't crash
during model construction.
@github-actions github-actions Bot added langchain `langchain` package issues & PRs size: L 500-999 LOC and removed size: M 200-499 LOC labels Mar 20, 2026
@github-actions github-actions Bot added size: M 200-499 LOC and removed size: L 500-999 LOC labels Mar 23, 2026
Mason Daugherty (mdrxy) and others added 2 commits March 22, 2026 22:58
Migrate all partner chat model classes from overriding the
`@model_validator`-based `_set_model_profile` to the new
`_resolve_model_profile` hook introduced in `langchain-core`. The old
pattern required each provider to duplicate boilerplate Pydantic
validator logic (`if self.profile is None: self.profile = ...`); the new
hook is a plain method that just returns a `ModelProfile | None`, with
the base class handling assignment.

## Changes
- Replace `_set_model_profile` `@model_validator(mode="after")`
overrides with `_resolve_model_profile() -> ModelProfile | None` across
10 partner packages: `anthropic`, `deepseek`, `fireworks`, `groq`,
`huggingface`, `mistralai`, `openai` (both `BaseChatOpenAI` and
`AzureChatOpenAI`), `openrouter`, `perplexity`, `xai`
- `ChatAnthropic._resolve_model_profile` preserves the `betas`-based
`max_input_tokens` override for the `context-1m-2025-08-07` beta — the
only provider with non-trivial profile resolution logic
- `AzureChatOpenAI._resolve_model_profile` and
`ChatHuggingFace._resolve_model_profile` retain their conditional guards
(`deployment_name is not None`, `model_id` exists) before attempting
lookup
@github-actions github-actions Bot added anthropic `langchain-anthropic` package issues & PRs deepseek `langchain-deepseek` package issues & PRs fireworks `langchain-fireworks` package issues & PRs github_actions Pull requests that update GitHub Actions code groq `langchain-groq` package issues & PRs huggingface `langchain-huggingface` package issues & PRs integration PR made that is related to a provider partner package integration and removed size: M 200-499 LOC labels Mar 23, 2026
@github-actions github-actions Bot added mistralai `langchain-mistralai` package issues & PRs openai `langchain-openai` package issues & PRs openrouter `langchain-openrouter` package issues & PRs perplexity `langchain-perplexity` package issues & PRs size: L 500-999 LOC xai `langchain-xai` package issues & PRs labels Mar 23, 2026
@mdrxy
Mason Daugherty (mdrxy) merged commit 2f64d80 into master Mar 23, 2026
226 of 227 checks passed
@mdrxy
Mason Daugherty (mdrxy) deleted the mdrxy/model-profiles/schema branch March 23, 2026 04:44
Michael Chin (michaelnchin) added a commit to langchain-ai/langchain-aws that referenced this pull request Mar 28, 2026
Follow-up to:
#959 (review)

With [this PR](langchain-ai/langchain#36129)
included in `langchain-core` v1.2.21, the following update has been made
for model profiles:
> Add BaseChatModel._resolve_model_profile() hook that returns None by
default. Partners can override this single method instead of redefining
the full _set_model_profile validator — the base validator calls it
automatically

Specifically, the old `_set_model_profile` validator defined by partner
chat models has been implemented directly in `BaseChatModel`, in favor
of the a new `_resolve_model_profile()` hook for partners to override.

This change breaks our current model profile resolution, but only for
AIPs. This is because `_set_model_profile` now runs before
`validate_environment` in the BaseChatModel's validator chain, meaning
that `_base_chat_model` will be run before the GetInferenceProfile API
call to retrieve the AIP's modelArn, so the raw AIP ARN string will be
passed for profile lookup instead (and subsequently return nothing).

To fix this, this PR:
- Replaces the `_set_model_profile` implementation in
`ChatBedrock`/`ChatBedrockConverse` with the new
`_resolve_model_profile` override.
- Additionally, for Converse, re-resolves `profile` at the end of
`validate_environment` for the AIP case where `base_model_id` isn't
available on the first pass.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

anthropic `langchain-anthropic` package issues & PRs core `langchain-core` package issues & PRs deepseek `langchain-deepseek` package issues & PRs fireworks `langchain-fireworks` package issues & PRs fix For PRs that implement a fix github_actions Pull requests that update GitHub Actions code groq `langchain-groq` package issues & PRs huggingface `langchain-huggingface` package issues & PRs integration PR made that is related to a provider partner package integration internal langchain `langchain` package issues & PRs mistralai `langchain-mistralai` package issues & PRs model-profiles `langchain-model-profiles` package issues & PRs openai `langchain-openai` package issues & PRs openrouter `langchain-openrouter` package issues & PRs perplexity `langchain-perplexity` package issues & PRs size: L 500-999 LOC xai `langchain-xai` package issues & PRs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant