Semantic DLP: Can AI Models Catch the Data Leak?

This is a submission for the Kaggle Benchmarking Challenge

What I Benchmarked

Employees paste astonishing things into AI tools: proprietary algorithms, unreleased
financials, incident postmortems, customer contract terms. Endpoint data-loss-prevention
(DLP) has to catch that before it leaves the machine, in milliseconds, on an
ordinary laptop, without sending the text anywhere.

I built a benchmark for the hard core of that problem: semantic leak detection.
Not regexes or known key formats (those are solved problems); the question is whether a model can tell
proprietary technical content from public technical content that looks nearly
identical.

The corpus is fully synthetic, needle-in-a-haystack style:

• 80 leaks (proprietary code, infrastructure configs, internal prose) generated with realistic mess (employee chat framing, paste artifacts, tuned constants, internal codenames), each buried at a random line boundary inside a haystack of benign technical text (4k–14k chars, mean ~9.6k)
• 140 pure negatives built from hard negatives: open-source-style code, public-doc tutorials, benign infra configs, casual engineering prose, the exact kind of text a naive classifier false-alarms on
• Every positive carries the char span of the buried leak, so the benchmark scores not just detection but localization (char-level IoU of the model's quoted span)

Two design choices worth stealing: hard negatives were written by the same generator
as the leaks (so a model can't win by recognizing writing style; it has to understand
content), and detection is scored at payload level on long mixed payloads, which is
what actually ships, not on clean isolated snippets.

Models Tested

Hosted LLMs via Kaggle Benchmarks (the leaderboard: free hosted inference, one
fixed prompt, structured verdict {is_leak, quote} for every model):

• google/gemini-2.5-pro and google/gemini-2.5-flash: the frontier ceiling
• openai/gpt-oss-120b (117B) and openai/gpt-oss-20b (21B): open-weight reasoning models

4 models × 220 payloads = 880 unique (model, payload) prompts, 1,200 task executions
(positives are prompted for both detection and quoting), 1,199 scored; the one
unparseable response was a quoting call on a positive payload, so catch rates
are unaffected and that model's localization mean is over 79 of 80. The suite reflects the Kaggle Benchmarks registry's current catalog;
the benchmark itself is model-agnostic, so newer or larger models slot in by
adding them to the runner.

The local-endpoint reference point (not on the kbench leaderboard; it runs on the
laptop, not an API): a frozen all-MiniLM-L6-v2 (22M params) with a logistic head
over 512-token windows (stride 128, payload decision = max window score), plus
ModernBERT-base (149M) and DeBERTa-v3-base (184M) as heavier comparators, evaluated on
a larger private set built the same way (3,000+ payloads, leakage-safe splits,
thresholds locked on a calibration split at ≥99% recall, 3 head seeds each).

Findings

Hosted rows: the public 220 payloads (95% Wilson intervals on false-alarm rates:
pro 32-48%, flash 15-28%, gpt-oss-120b 12-24%, gpt-oss-20b 17-31%). Local rows:
the private 3,000+ payload eval, where the ± is spread over head seeds,
not over which negatives were drawn; the follow-up section reports the local
model on the same 220 payloads (catch 1.000, false alarms 61.7%).

* identical shared Kaggle CPU VM, INT8 dynamic-quantized PyTorch, 100 payloads of
~26 windows each; treat as a relative comparison (ONNX Runtime on a real laptop is
several × faster in absolute terms).

The local-side story (measured, 3 seeds, 2%-prevalence eval):

• A 22M-parameter embedding model beat a 149M and a 184M encoder at their own game: perfect recall on buried leaks at 3–5× lower false-alarm rate (per-window false-alarm ~0.3% vs 0.9% / 1.9%). Contrastively-trained sentence embeddings are linearly separable for this task; raw language-model features aren't. When your deployment plan is a frozen encoder + tiny head (cheap to ship, seconds to retrain), embedding- tuned models punch far above their parameter count.
• DeBERTa-v3 is the alarm-fatigue cautionary tale: near-top recall (97.5%) but a 36% payload false-alarm rate: it fires on a third of benign technical pastes. In DLP, the tool that keeps raising false alarms gets uninstalled; recall alone would have picked the wrong model.
• Burying a leak doesn't hide it: detection of needles buried in 14k-char haystacks matched or beat detection of the same leaks standalone (dilution ≈ 0 to negative for all three models). Long-context dilution is not the failure mode people assume.
• At 2% prevalence, FDR is the honest headline: even the winner's false discoveries outnumber true catches (~4:1 at a 99%-recall threshold tuned from only 60 calibration positives, a deliberately conservative protocol). Thresholds must be tuned on a calibration split, never the eval set; and 1-in-5 vs 1-in-11 alert precision is the difference between a tool people trust and one they disable.

The LLM-side story (4 hosted models, 879 scored runs, single fixed prompt):

• Catch rate tracks model class: gemini-2.5-pro catches every buried leak (1.000), flash misses 4% (0.962), gpt-oss-120b 17% (0.825), and the 21B gpt-oss-20b misses 1 in 4 (0.750). The frontier-vs-open-weight gap is statistically solid (77/80 vs 66/80, Fisher p = 0.009); the 117B-vs-21B step is not separable at n=80 (p = 0.33), and 80/80 only bounds the frontier's rate at 96% or better.
• The recall champion is the false-alarm champion: gemini-2.5-pro flags 39.3% of clean hard negatives, worse than DeBERTa-v3's 36%, the local side's cautionary tale. No hosted model got below 17.1% on this set; the local 22M model runs at 7.3% on its private eval (cross-set: see the follow-up section below) (caveat: the LLMs sit at one prompt-implied operating point, while the encoder's threshold is calibrated to 99% recall on a held-out split; the comparison favors the encoder in fairness terms, and it still wins).
• Only the frontier model can point at the leak: gemini-2.5-pro quotes it nearly verbatim (IoU 0.85); flash is mediocre (0.39); the gpt-oss models flag the payload but can't produce a usable verbatim span (0.036–0.060; strict scoring: a paraphrased quote earns nothing). An alert UI needs the span, not just the bell.

The thesis the numbers support: hosted LLMs reach the recall ceiling but are
structurally wrong as the endpoint layer: each payload costs seconds of network round-trip and per-call fees,
and using them means transmitting the exact text under suspicion off the device, which
is the behavior a DLP product exists to prevent. On-device and free remain true; whether the local encoder also wins on accuracy is distribution-dependent, and the follow-up section below measures both sides of that.

Follow-up: the local model on the same 220 payloads

The local 7.3% false-alarm rate and the hosted 17-39% come from different eval
sets (private 3,000+ payloads vs the public 220), so as a direct control I ran
the encoder on the same public payloads, head retrained after excluding every
220 source from training (mandatory: all 80 public needles also appear in the
private train split).

On the same payloads the ordering inverts: the 22M model still catches everything
but false-alarms on nearly two thirds of clean payloads.

Why: the public 220's benign text is LLM-written (296 of the 300 generator
chunks), while the private train and calibration sets contain zero LLM-written
payloads; the head had simply never seen that writing style as negative. The length
dependence of max-over-windows aggregation is also visible in isolation: false
alarms climb
42% / 65% / 84% across payloads of up to 10 / 11-20 / 21-30 windows, consistent
with max-over-windows aggregation giving longer pastes more chances to trigger.

A natural control is to retrain with half of the LLM-written negatives added to
training and measure the held-out half: false alarms drop from 53% to 20% at
unchanged 1.000 recall. But this cannot be read as a fix. The two halves share
156 of 221 generator chunks (69 of 70 held-out payloads contain text the head saw
in training), so the improvement is consistent with memorizing seen chunks.
Whether fresh, unseen generator-written negatives would repair the small model is
an open question: the public set consumed 296 of the 300 available chunks, so a
clean answer requires generating a new negative pool. What the same-set result
does establish: hard negatives must match the deployment distribution, and a
private eval whose negatives differ in generator from the public ones can invert
a model ranking.

Limitations, stated plainly: synthetic corpus (no real employee data, by design); hosted-model rates are
single-run and stochastic (a full repeat of the suite moved catch/false-alarm rates by
up to 7pp; rank order held);
80/140 payloads per arm for the LLM suite, 100 eval leaks for the encoders (recall CI
±5–8pp); localization scoring is strict (verbatim quote must appear in the payload);
encoder heads are linear probes on frozen features (a floor, not a ceiling;
fine-tuning would likely close some gaps); encoder latency measured on a shared Kaggle
CPU VM with an unoptimized backend (relative comparison only).

My Benchmark

Semantic DLP: Catch the Data Leak: the payload set is attached and fully
synthetic, so anyone can re-run it, add models, or fork the three tasks
(leak_catch_rate / stay_silent_rate / locate_leak). The public leaderboard initially lists the models published through the
platform's single-run output; the full four-model suite results are in the table
above and published in the run output at
https://www.kaggle.com/code/karthiksethuraman6/leak-catch-rate/output : the summary
dlp_llm_summary.csv and the backup_runs/ folder with every model's raw run
records are all there. The board extends via Add Models.
One harness note for reproducers:
reasoning models (gpt-oss) wrap their output in <think> blocks, so the verdict parser
extracts the last JSON object from the raw response rather than trusting strict
structured-output parsing.

Next steps: span-level heads on the winning encoder (the corpus already labels the
leak's exact characters; classification is only half of a usable alert); a second
operating point at 95% recall for a two-point precision/recall curve; conformal
thresholds; and extending the corpus to more leak categories.