fix(core,model-profiles): add missing ModelProfile fields, warn on schema drift - #36129
Conversation
Merging this PR will degrade performance by 65.01%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | WallTime | test_async_callbacks_in_sync |
20.5 ms | 15.1 ms | +35.57% |
| ❌ | WallTime | test_init_time |
2.8 ms | 3.2 ms | -12.41% |
| ❌ | WallTime | test_init_time |
2.8 ms | 3.2 ms | -11.28% |
| ⚡ | WallTime | test_init_time |
140.8 ms | 124.8 ms | +12.85% |
| ❌ | Simulation | test_init_time |
652.1 µs | 1,863.5 µs | -65.01% |
| ❌ | Simulation | test_init_time_with_client |
1.8 ms | 3 ms | -39.87% |
Comparing mdrxy/model-profiles/schema (337e788) with master (5ffece5)
Footnotes
-
15 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
Wrap `self.profile` access in `_set_model_profile` and `_check_profile_keys` with try/except AttributeError so subclasses that override `__getattribute__` to raise on "profile" don't crash during model construction.
Migrate all partner chat model classes from overriding the `@model_validator`-based `_set_model_profile` to the new `_resolve_model_profile` hook introduced in `langchain-core`. The old pattern required each provider to duplicate boilerplate Pydantic validator logic (`if self.profile is None: self.profile = ...`); the new hook is a plain method that just returns a `ModelProfile | None`, with the base class handling assignment. ## Changes - Replace `_set_model_profile` `@model_validator(mode="after")` overrides with `_resolve_model_profile() -> ModelProfile | None` across 10 partner packages: `anthropic`, `deepseek`, `fireworks`, `groq`, `huggingface`, `mistralai`, `openai` (both `BaseChatOpenAI` and `AzureChatOpenAI`), `openrouter`, `perplexity`, `xai` - `ChatAnthropic._resolve_model_profile` preserves the `betas`-based `max_input_tokens` override for the `context-1m-2025-08-07` beta — the only provider with non-trivial profile resolution logic - `AzureChatOpenAI._resolve_model_profile` and `ChatHuggingFace._resolve_model_profile` retain their conditional guards (`deployment_name is not None`, `model_id` exists) before attempting lookup
Follow-up to: #959 (review) With [this PR](langchain-ai/langchain#36129) included in `langchain-core` v1.2.21, the following update has been made for model profiles: > Add BaseChatModel._resolve_model_profile() hook that returns None by default. Partners can override this single method instead of redefining the full _set_model_profile validator — the base validator calls it automatically Specifically, the old `_set_model_profile` validator defined by partner chat models has been implemented directly in `BaseChatModel`, in favor of the a new `_resolve_model_profile()` hook for partners to override. This change breaks our current model profile resolution, but only for AIPs. This is because `_set_model_profile` now runs before `validate_environment` in the BaseChatModel's validator chain, meaning that `_base_chat_model` will be run before the GetInferenceProfile API call to retrieve the AIP's modelArn, so the raw AIP ARN string will be passed for profile lookup instead (and subsequently return nothing). To fix this, this PR: - Replaces the `_set_model_profile` implementation in `ChatBedrock`/`ChatBedrockConverse` with the new `_resolve_model_profile` override. - Additionally, for Converse, re-resolves `profile` at the end of `validate_environment` for the AIP case where `base_model_id` isn't available on the first pass.
PR #35788 added 7 new fields to the
langchain-profilesCLI output (name,status,release_date,last_updated,open_weights,attachment,temperature) but didn't updateModelProfileinlangchain-core. Partner packages likelangchain-awsthat setextra="forbid"on their Pydantic models hitextra_forbiddenvalidation errors when Pydantic encountered undeclared TypedDict keys at construction time. This adds the missing fields, makesModelProfileforward-compatible, provides a base-class hook so partners can stop duplicating model-profile validator boilerplate, migrates all in-repo partners to the new hook, and adds runtime + CI-time warnings for schema drift.Changes
langchain-core__pydantic_config__ = ConfigDict(extra="allow")toModelProfileso unknown profile keys pass Pydantic validation even on models withextra="forbid"— forward-compatibility for when the CLI schema evolves ahead of coreModelProfile:name,status,release_date,last_updated,open_weights(metadata) andattachment,temperature(capabilities)_warn_unknown_profile_keys()inmodel_profile.py— emits aUserWarningwhen a profile dict contains keys not inModelProfile, suggesting a core upgrade. Wrapped in a bareexceptso introspection failures never crash model constructionBaseChatModel._resolve_model_profile()hook that returnsNoneby default. Partners can override this single method instead of redefining the full_set_model_profilevalidator — the base validator calls it automaticallyBaseChatModel._check_profile_keysas a separatemodel_validatorthat calls_warn_unknown_profile_keys. Uses a distinct method name so partner overrides of_set_model_profiledon't inadvertently suppress the checklangchain-profilesCLI_warn_undeclared_profile_keys()to the CLI (cli.py), called after merging augmentations inrefresh()— warns at profile-generation time (not just runtime) when emitted keys aren't declared inModelProfile. Gracefully skips iflangchain-coreisn't installedtest_model_data_to_profile_keys_subset_of_model_profilein model-profiles — feeds a fully-populated model dict to_model_data_to_profile()and asserts every emitted key exists inModelProfile.__annotations__. CI fails before any release if someone adds a CLI field without updating the TypedDictPartner packages
_resolve_model_profile()hook, replacing duplicated@model_validator/_set_model_profileoverrides: anthropic, deepseek, fireworks, groq, huggingface, mistralai, openai (base + azure), openrouter, perplexity, xaimax_input_tokensoverride); all others reduce to a one-linerpr_lint.ymlscope for the newmodel-profilespackage