This is a submission for the Kaggle Benchmarking Challenge

What I Benchmarked

TL;DR. I tested 17 AI models on writing, recalling and fixing OpenTelemetry instrumentation, graded in code against 26 releases of the spec.

• 16 of 17 models wrote retired names, and 83% of those names had been retired for more than a year.
• 12 of 13 models wrote names they had correctly replaced when asked directly.
• 7 of 17 "fixed" a metric by changing its unit and keeping its retired name, which breaks it both ways.
• One sentence naming the spec version cut retired names by 36%, and made models invent more.

Benchmark: Semconv Drift on Kaggle

I asked Claude Sonnet 5 what replaced the OpenTelemetry attribute db.system. It answered correctly: db.system.name, the name the spec has used since January 2025.

In the same benchmark, I asked it to add tracing to a go-redis call. It wrote this:

attribute.String("db.system", "redis"),
attribute.String("db.operation", cmdName),
attribute.String("db.statement", statement),

All three names are retired. The code compiles, the span exports, the backend stores it. Nothing fails. Weeks later, a dashboard that filters on db.system.name shows no Redis traffic, and nobody connects the empty panel to a line of generated code.

This year I have had 14 pull requests merged into OpenTelemetry's Go compile-time instrumentation project, and this is exactly the kind of bug I would wave through in review. The code looks right. It is only wrong against a spec version, and nobody reviews against a spec version. So I built a benchmark to measure how often it happens.

The task

OpenTelemetry's semantic conventions are the shared vocabulary of telemetry: which key holds the HTTP method, what a database span calls its query, which unit a duration metric uses. They are versioned, and many of the names people learned first have been renamed. http.method became http.request.method in 2023. db.statement became db.query.text in 2024. rpc.client.duration, in milliseconds, became rpc.client.call.duration, in seconds, in January 2026.

Semconv Drift has four tasks on Kaggle Benchmarks:

Every Write prompt already says "Follow the current OpenTelemetry semantic conventions." The models were told. The question is what "current" means to them.

Grading is code, not a judge. The grader extracts every semantic convention name from an answer (string keys, metric names, SDK constants in four languages, enum values) and looks each one up in 1,563 names from 26 releases of the spec, v1.21.0 to v1.44.0, plus every name from v1.0.0 to v1.20.0 for the oldest ones. Every retired name carries the release and date that retired it. Reasoning text is stripped first, so a model is judged on the code it wrote. The answer key is tested too: the build fails if any gold answer is not a current name in the ground truth.

The ground truth comes from Attrition, a dataset of every tagged release of open-telemetry/semantic-conventions that I built for DEV's Sanity challenge.

Models Tested

17 models through Kaggle's model proxy, picked to answer three questions:

• Does a newer generation fix it? Three same-family pairs: Claude Sonnet 4.5 and 5, GPT-5.4 and 5.5, Gemini 2.5 Pro and 3.1 Pro.
• Does spending more fix it? Frontier models (GPT-6 Astra, Claude Opus 5, GPT-5.5, Gemini 3.1 Pro) next to small, cheap ones (GPT-5.4 mini, Claude Haiku 4.5, Gemini 3.7 and 3.8 Flash).
• Do open-weight models differ? Qwen 3 235B, DeepSeek-R1, GLM-5, Gemma 4 31B and gpt-oss-120b.

That is 830 graded answers, plus 84 more from repeat runs. Four reasoning models (DeepSeek-R1, GLM-5, Gemma 4 and gpt-oss-120b) have no Know score: on some short factual questions they spent the whole answer budget thinking and never answered. Grok 4.5 and 4.6 returned 404 for my account, so they are not in the lineup.

Findings

1. The drift is old, not new

16 of 17 models wrote at least one retired name in Write, 357 uses in all. If this were about training cutoffs, the retired names would be recent ones the models never saw. They are not:

• 297 of 357 (83%) had been retired for more than a year when I ran the benchmark.
• 238 (67%) had been retired for more than two years, which means before October 2024. Every model in the lineup was released after that.
• The median retired name was retired 2.4 years earlier.

These models were not missing the news. They had read the changelog, and years of tutorials before it, and the tutorials won. db.system is the clearest case: 24 uses across 12 models, nearly two years after the rename.

The only names a training cutoff could excuse are the January 2026 rpc.* renames. 16 models wrote them. GPT-6 Astra was the only exception, and it also avoided every older retired name, so a late cutoff alone does not explain its perfect Write score.

2. It told me the new name, then wrote the old one

Because Know and Write ask about the same names, I can check each model against itself. For every retired name a model wrote in Write, did the same model give the correct replacement in Know?

12 of 13 models wrote at least one retired name they had correctly replaced when asked directly, 58 times in all. Claude Haiku 4.5 said http.method became http.request.method, then wrote http.method five times. Sonnet 5 did it with db.system, as in the opening. Only GPT-6 Astra never did it.

This is not a knowledge gap. Asked a question, a model retrieves the changelog. Asked for code, it retrieves the most common code, and the most common code is old.

3. The half fix

One Fix case hands the model this Go snippet and asks it to move to v1.44.0, converting values if a unit changed:

dur, _ := meter.Float64Histogram("rpc.client.duration", metric.WithUnit("ms"))
start := time.Now()
err := invoker(ctx, method, req, reply, cc, opts...)
dur.Record(ctx, float64(time.Since(start).Milliseconds()))

The answer is rpc.client.call.duration, unit s, values in seconds. Only GPT-6 Astra got all of it.

7 of 17 models changed the unit to seconds, converted the values, and kept the retired name. That is worse than leaving the code alone. Unchanged code is at least consistent with the old spec. The half-fixed metric emits the old name with the new unit, so it breaks in both directions: queries on the old name read values 1,000 times smaller than expected, and queries on the new name find nothing. The diff looks like a fix, so it passes review.

Two more models, from two different vendors, did something stranger. GPT-5.5 and Gemini 3.7 Flash both renamed the metric to rpc.client.request.duration. No release of the spec has ever defined that name. It is a plausible guess built from the real http.client.request.duration, and both models made the same guess.

4. Why this domain drifts when others do not

Another entry in this challenge tested removed pandas and Express APIs and found that models used the current ones. I think the difference is how each change fails.

A removed function fails loudly. Call DataFrame.append on pandas 2 and you get an error, a Stack Overflow question and an answer, and the answer ends up in training data. A retired attribute name fails silently. Nothing throws, nobody asks, and the old tutorials are never corrected. The OpenTelemetry project knows this: its instrumentation libraries offer OTEL_SEMCONV_STABILITY_OPT_IN=http/dup, which emits both the old and the new HTTP names during a migration, because a rename does not propagate on its own.

That is a hypothesis, not a measurement. But it predicts where to look for drift: anywhere the old form still works.

5. One sentence of context helps, and it is not enough

The hint task repeats the 12 Write requests with one added sentence: "Target OpenTelemetry semantic conventions v1.44.0 (released August 2026). Many attribute and metric names were renamed or removed between 2023 and 2026, so do not rely on older names."

Before trusting any difference, I needed a noise floor. Kaggle's benchmark library accepts a temperature but drops it unless a model is flagged as supporting it, so the same prompt can give a different answer twice. I ran the plain Write task a second time on seven models. Their retired-name total moved from 156 to 166, about 6%, with nothing changed. The hint moved the same seven models from 166 to 94, a 43% drop.

Across all 17 models:

• Retired names fell by 36%, from 357 to 230. The mean Write score rose from 39% to 48%.
• Only one model reached zero, and it was the one that was already there.
• Invented names rose from 21 to 31. Told that names had changed, some models stopped reaching for old names and started guessing new ones. Claude Haiku 4.5 went from 1 invented name to 7.

Another entry in this challenge found that a one-sentence rule took clean answers from 64% to 96%. Here one sentence removed about a third of the drift and replaced part of it with guesses. A version number tells a model that names changed. It does not tell it what they changed to.

6. The leaderboard

Overall is the average of a model's task scores, the same rule the Kaggle leaderboard uses.

Two patterns are weaker than the findings above. Two of the three family pairs improved clearly (Sonnet 4.5 to 5: 52% to 77%; Gemini 2.5 Pro to 3.1 Pro: 26% to 51%), and the GPT pair tied (GPT-5.4 76%, GPT-5.5 75%). Price was a poor guide: GPT-5.4 matched GPT-5.5 for under a tenth of the cost ($0.10 against $1.36).

What surprised me

I expected a cutoff story: new renames missed, old ones known. I got the opposite. The models know the renames, and they still write the names from the tutorials they read most. The half fix surprised me more. I wrote that case to catch models that forget the unit. Almost none forgot the unit. They forgot the name.

What this changed about how I review AI code

I used to treat "the model knows the current API" as the thing to check. It is the wrong check. Asking a model what changed tells you nothing about what it will write. For telemetry code, I now check the names against the spec, not against my memory, and I treat a diff that changes a unit but not a name as a bug.

What this means for you

• Lint telemetry names in CI. The spec's registry marks every deprecated name and its replacement. A check of string literals against that list would flag every retired name counted in this post, db.system included.
• Put the spec version in the prompt, then still lint. It cut drift by a third here, and it also made models invent names. If you can, paste the current names you need instead of the version number.
• Prefer generated constants pinned to a version. In Go the semconv version is in the import path. Bump it on purpose.
• Review units and names together. A histogram whose unit changed and whose name did not is the hardest drift to spot, because the diff looks like a fix.

Language: Python.
Wrap this psycopg query in an OpenTelemetry client span. Record which database system it is, the database name, the operation, the table and the query text.

def list_orders(conn, customer_id):
    with conn.cursor() as cur:
        cur.execute('SELECT * FROM orders WHERE customer_id = %s', (customer_id,))
        return cur.fetchall()

Follow the current OpenTelemetry semantic conventions. Write attribute keys and metric names as string literals rather than SDK constants. Return one complete code block and nothing else.

grep -nE "[\"'](db\.system|db\.name|db\.operation|db\.statement|db\.sql\.table|net\.peer\.name)[\"']" answer.py

What I would measure next

• More repeats. Seven models ran Write twice. Every model should run every task several times, with a reported spread.
• Names instead of a version. Does pasting the current names into the prompt remove the drift and the guessing, or only the drift?
• Units beyond time. The grader checks ms against s only. Byte and count units can drift too.
• The GenAI conventions. They moved to their own repository and are changing fast, which makes them the next place to look.

Limitations

• Small case counts. Write has 12 cases, Know 22 and Fix 8. Claude Sonnet 5's 67% on Write has a 95% Wilson interval of 39% to 86%. The repeat runs flipped up to 3 of 12 Write cases for one model, so I would not claim any order in the middle of the table.
• Know is missing for four reasoning models, which ran out of answer budget while thinking.
• Grok is missing. Grok 4.5 and 4.6 returned 404 through Kaggle's model proxy.
• One strict call. Claude Opus 5 lost one Write case only for naming a custom key app.ledger.entry_count. The spec has owned the app.* namespace since its mobile conventions, so the grader counts it as invented, and the rule applies to every model the same way.
• A moving target. Each semconv release retires more names. Re-running after the next release needs only a new ground-truth export, which is also what keeps this benchmark from going stale itself.

My Benchmark

Kaggle benchmark: https://www.kaggle.com/benchmarks/applesone/semconv-drift

The four tasks, each with its own leaderboard and the full task code:

Interactive results: a sortable leaderboard with confidence intervals and every chart, at https://vignesh2027.github.io/semconv-drift/

Code, grader and ground truth:

analyze.py and charts.py regenerate every number and chart in this post from the Kaggle runs. If you maintain OpenTelemetry instrumentation and want a case added, open an issue on the repo.