KCC Negative Innovation Deadlock: Why N=3 Guarantees Convergence

KCC Negative Innovation Deadlock: Why N=3 Guarantees Convergence

0. Parameters and Notation

All calculations below are based on KCC default parameters, cross-verified in the source code (tcp_kcc.c):

Derived constants:

$$\begin{array}{rl}converged\_val& =\frac{250000\cdot 400}{1000000-250000}-100=\frac{10^8}{7.5\times 10^5}-100=33\\ dyn\_thresh& =5\text{ms}\cdot 1000\text{(μs/ms)}\cdot 1024=5.12\times 10^6(\text{approx. 5ms × scale})\\ q\_boost\_thresh& =4\cdot 1\text{ms}\cdot 1000\cdot 1024=4.096\times 10^6(\text{approx. 4ms × scale})\end{array}$$

Scenario: Chengdu → Hong Kong, normally routed through Shanghai (long path 65ms), occasionally flopping to Guangdong (short path 45ms).

Scaled physical quantities ($\times 1024$):

$$x\_est(\text{65ms})=66560,z(\text{45ms})=46080,\nu =46080-66560=-20480$$

Drop ratio = $(65-45)/65=30.8\%$, far exceeding the floor gate’s upper bound of 12.5%.

1. Old Code: Per-RTT Derivation (Why It Was Unreliable)

The old code (commit 95f28e7) had the following gating conditions:

Outlier gate: |ν| > dyn_thresh && p_est ≤ converged_val
Floor gate:   p_est > converged_val || z ≥ x_est × 7/8

Case A: $p\_est=10$ (filter at floor value, Q suppressed by a long-term clean path)

RTT 1: $z=46080$, $\nu =-20480$, $|\nu |=20480$

① Outlier gate: $20480>5.12\times 10^6$ ✓, $p\_est=10\le 33$ ✓ → Reject

Upon rejection, $p\_est$ is set to $p\_pred$. The filter state is updated (even if the sample is rejected):
$$p\_pred=p\_est+Q=10+100=110$$
$$p\_est:=p\_pred=110$$

The rejected sample does not enter $x\_est$:
$$x\_est=66560(\text{unchanged, still 65ms})$$

② The function returns, never reaching the floor gate.

RTT 2: $z=46080$, $\nu =-20480$

① Outlier gate: $p\_est=110>33$ → $p\_est\le 33=false$ → Not triggered

② Sample bypasses the outlier gate and enters the floor gate:
$$p\_est=110>33\to \text{short-circuit condition triggers}\to \text{floor skipped}$$
$$x\_est=z=46080(\text{45ms, converged!})$$

Case A conclusion: With $Q=100$, after RTT 1 rejects the sample, $p\_est$ rises to 110, and RTT 2 bypasses the floor. Converged in 2 RTTs, no deadlock.

Case B: $p\_est=10$, $Q=0$ (extreme suppression – lower bound of adaptive Q)

RTT 1: $z=46080$, $\nu =-20480$

① Outlier gate: $20480>5.12\times 10^6$ ✓, $10\le 33$ ✓ → Reject

With $Q=0$: $p\_pred=p\_est+Q=10+0=10$, $p\_est:=10$ (unchanged!)

RTT 2: $p\_est=10\le 33$ ✓ → Outlier gate triggers again → Reject.

$p\_pred=10+0=10$, $p\_est:=10$ (still unchanged!)

RTT 3: same as above → Reject.

RTT N: $p\_est=10$ never rises → Outlier gate triggers forever → sample never reaches the floor gate → $x\_est$ stuck at 65ms forever.

Case B conclusion: With $Q=0$, the filter is permanently deadlocked.

The Fundamental Problem

The behavior of the old code depended on the value of $Q$ — and $Q$ is dynamically adaptive (depending on min_rtt_us / q_rtt_div), not a constant the designer can guarantee in advance. It is impossible to assert “this code will converge under all conditions.”

2. New Code: Per-RTT Derivation (Deterministic Proof)

The new code (commit 88f1b8e) completely replaced the $p\_est$ gating for negative innovations — using the neg_skip_cnt persistence counter instead. The following derivation does not depend on any specific values of $p\_est$, $Q$, or $R$.

Assumptions (worst case):

• $x\_est=66560$ (65ms, steady-state value of the long path)
• $z=46080$ (45ms, measurement from the new short path, constant every RTT)
• $neg\_skip\_cnt=0$ (starting state, no prior negative innovation)
• $last\_neg\_mstamp=0$ (negative timestamp never recorded)

RTT 1

① neg_skip_cnt increment

if (innovation < 0) {
    u32 threshold = rtt_us >> 1;    // 45000/2 = 22500
    threshold = max_t(u32, threshold, 1U);
    // = 22500
    bool enough_time_elapsed;
    enough_time_elapsed = (0 == 0) || ((now - 0) >= 22500);
    // last_neg_mstamp == 0 → true
    if (enough_time_elapsed) {
        ext->neg_skip_cnt = (0 < 254) ? 0+1 : 254;
        // = 1
    }
}

neg_skip_cnt: 0 → 1. last_neg_mstamp = now.

② Outlier gate

if (unlikely(!qboost_fired && abs_innov > dyn_thresh)) {
    // 20480 > 5.12e6 → true
    if (innovation < 0) {
        outlier_reject = (1 < 3);  // true
    }
    if (outlier_reject) {
        // consec_reject_cnt(0) < 25 → true
        // p_est = p_pred, update jitter, return
    }
}

$outlier\_reject=true$ → Reject.

Side effects on rejection: $p\_est:=p\_pred=p\_est+Q$ ($p\_est$ increases, but this has no effect on the new gating — the outlier gate no longer inspects $p\_est$).

③ x_est unchanged: $x\_est=66560$ (65ms).

RTT 2

① neg_skip_cnt increment

$now-last\_neg\_mstamp\approx 45000\mu s$ (one RTT ≈ 45ms)

$$45000\ge 22500\to enough\_time\_elapsed=true$$

$$neg\_skip\_cnt=1+1=2$$

② Outlier gate

$$outlier\_reject=(2<3)=true\to \text{Reject}$$

③ x_est unchanged: $x\_est=66560$ (65ms).

RTT 3

① neg_skip_cnt increment

$$45000\ge 22500\to enough\_time\_elapsed=true$$

$$neg\_skip\_cnt=2+1=3$$

② Outlier gate

$$outlier\_reject=(3<3)=false\to \text{Bypass!}$$

The sample penetrates the outlier gate and enters subsequent processing.

③ Directional gate: $\nu =-20480<0$ → $\nu \le 0$ branch → negative innovation → x_est = z path.

④ Floor gate

u64 floor = x_est - (x_est >> 3);
// floor = 66560 - 8320 = 58240

if (neg_skip_cnt >= 3 || z >= floor) {
    // 3 >= 3 → true! Bypass the floor check!
    x_est = (u32)min_t(u64, z, U32_MAX);
    // = 46080 (45ms!)
}

⑤ x_est = 46080 (45ms) ← Converged!

⑥ Automatic reset after convergence:

if (x_updated) {
    ext->pos_skip_cnt = 0;
    ext->neg_skip_cnt = 0;  // ← Cleared
    ext->drift_sum = 0;
}

$neg\_skip\_cnt=0$, ready for the next potential route change.

RTT 4: Steady-state maintenance

• $x\_est=46080$ (45ms), $z\approx 46080$, $\nu \approx 0$
• $\nu \approx 0$ → $\nu \ge 0$ branch → neg_skip_cnt = 0 (cleared)
• $|\nu |\approx 0<5.12\times 10^6$ → Outlier gate does not trigger
• Floor gate: $z\approx 46080\ge 46080\times 0.875=40320$ ✓ → passed normally
• Normal Kalman update maintains $x\_est$ around 45ms

Convergence summary table

Key property: The entire chain does not check $p\_est$, $Q$, or $R$. Convergence depends solely on the independent state neg_skip_cnt.

3. Why N=3: Joint Probability Under Multi-Layer Defense

neg_skip_cnt is just a counter — whether it can correctly increment to 3 depends on whether the preceding samples can penetrate all front-line defense layers.

3.1 Sources of False Negative Innovations

Three known sources of ACK contamination can produce spurious $\nu <0$:

3.2 A Single Layer is Insufficient: A Simple Probability Estimate

Consider only the outlier gate’s base threshold (5ms): on a 500Mbps link, TSO-induced RTT error ≈ 1ms < 5ms → a single TSO batch will not trigger the outlier gate. However, on a 100Mbps link, the TSO error can reach 5.1ms — just crossing the threshold.

If only the outlier gate existed, a single TSO event on a slow link could produce a spurious rejection. But neg_skip_cnt requires 3 consecutive times to break through — requiring TSO batches over 3 independent RTTs, each crossing the 5ms threshold.

From a naive binomial perspective: $2^{-3}=12.5\%$ is already a conservative upper bound. But the true probability is far lower because four pre-defense layers are active simultaneously.

3.3 Joint Filtering by Four Defense Layers

For a spurious negative to successfully enter neg_skip_cnt, it must simultaneously penetrate all of the following layers:

RTT sample → [Temporal gating] → [Outlier gate base threshold] → [Jitter EWMA dynamic threshold] → [Directional gate] → neg_skip_cnt++

First Layer: Temporal Gating ($rtt/2$ interval)

ACK spacing within a TSO batch is about 1μs; $rtt/2=22.5$ms. Only the first ACK of the first batch passes. The subsequent ~43 ACKs are all discarded by the temporal gate.

$$P_{time}=\frac{1}{44}\approx 0.023(\text{a single TSO batch yields only 1 count opportunity})$$

Second Layer: Outlier Gate Base Threshold (5ms)

Triggers only when $|\nu |>5$ms. TSO-induced RTT error = $64\text{KB}/C$:

On the 500Mbps link discussed in this article, the TSO error itself cannot trigger the outlier gate. The outlier gate responds to TSO only on truly large path switches (20ms step) or extremely slow links.

Third Layer: Jitter EWMA Dynamic Threshold

Even if the outlier gate is triggered by the base threshold, a dynamic threshold applies:

$$dyn\_thresh=\max (outlier\_ms\cdot scale,jitter\_ewma\cdot outlier\_jitter\_mult\cdot scale)$$

$jitter\_ewma$ is an EWMA of jitter with $\alpha =0.125$ ($N_{eff}\approx 15$ samples). An isolated RTT spike caused by TSO is diluted by a factor of $1/8$ in the EWMA — a single spurious pulse hardly raises the dynamic threshold.

When $jitter\_ewma$ is low, the dynamic threshold degrades to the base threshold (5ms). But when real sustained jitter exists on the path, the dynamic threshold is raised, making subsequent TSO artifacts even harder to trigger the gate.

Fourth Layer: Directional Gate

Samples with $\nu >0$ are rejected by the directional update rule. However, TSO/GSO happen to produce $\nu <0$ (RTT biased low) — this bypasses the directional gate.

But there is another crucial fact: ACK compression not only produces $\nu <0$, but also produces temporal correlation: negatives from the same TSO batch are clustered in time (μs-scale spacing), and the temporal gate is specifically designed to counter this clustering.

3.4 Joint Probability

On the 500Mbps × 45ms RTT link discussed here:

$$P(\text{single-RTT false negative})=P_{time}\cdot P_{base}\cdot P_{jitter}\cdot P_{directional}$$

$$=0.023\cdot \approx 0\cdot 1.0\cdot 1.0\approx 0$$

TSO error ≈ 1ms on a 500Mbps link, far below the 5ms base threshold — the second defense layer already pushes the probability to zero.

On a 100Mbps link (TSO error ≈ 5.1ms, marginally passing the base threshold):

$$P(\text{single-RTT false negative})\approx 0.023\cdot 0.5\cdot 1.0\cdot 1.0\approx 0.012$$

$$P(\text{3 consecutive RTTs})\approx 0.012^3\approx 1.7\times 10^{-6}$$

This is about 1 in 600,000 — three independent filters (temporal gating × base threshold × EWMA smoothing) suppress the penetration rate of a single TSO artifact to ~1.2%, and the joint probability of three penetrations to the parts-per-million level.

Conversely, a real physical path drop (e.g., BGP reroute) produces a stable $\nu <0$ every RTT; both the base and dynamic thresholds are triggered consistently, and the temporal gate is satisfied by the regular intervals. Real signals and TSO artifacts are highly distinguishable under multi-layer joint filtering.

3.5 Re-examining N=2 vs N=3

From the joint probability perspective:

N=3 is not determined by a simple $2^{-3}$, but by the actual penetration probability under the joint action of multiple defense layers. $2^{-3}$ is merely a theoretical loose upper bound — in real engineering, the four pre-filters have already suppressed the single-RTT false negative probability to ~1.2%, and the three-consecutive joint probability to below one in a million. Even if this one-in-a-million occurs, the persistent upward pull provided by drift detection and Q-boost (see §3.7) will correct $x\_est$ back to the true $T\_prop$ within tens to hundreds of RTTs. The asymmetric design of “fast downward pull by negatives + slow upward pull by positives” guarantees the filter’s robustness across all timescales.

3.6 Marginal Analysis of N=4

If KCC_NEG_PERSIST_THRESH were set to 4, the convergence process would become:

Convergence latency: $4\times 45\text{ms}=180\text{ms}$.

Joint probability for N=4 on a 100Mbps link:

$$P(\text{4 consecutive RTT false negatives})\approx 0.012^4\approx 2.1\times 10^{-8}$$

Compared to N=3’s $1.7\times 10^{-6}$, N=4 reduces the false-positive rate by two orders of magnitude — from one in a million to one in a hundred million.

But this comes with a real additional latency cost: N=3 → N=4 adds 1 RTT (45ms) of convergence waiting. In the Neyman-Pearson marginal analysis:

From N=3 to N=4, the marginal probability gain is only $1.7\times 10^{-6}$ per RTT — nearly two orders of magnitude smaller than the N=2→3 gain of $1.4\times 10^{-4}$ per RTT. Waiting an extra RTT yields safety improvement that is statistically dwarfed, yet every real path switch of the same duration is delayed by one more RTT. This is classic diminishing marginal returns — N=3 is the last node on the curve with a still-significant marginal value.

3.7 The Ultimate Safety Net: Even if the One-in-a-Million Occurs, Drift + Q-boost Will Pull It Back

Assume the extreme scenario — the one-in-a-million probability actually happens — three consecutive TSO/GSO artifacts penetrate all four defense layers, and $x\_est$ is incorrectly pulled down to a value lower than the true $T\_prop$ (e.g., 40ms while the true path is 45ms).

What happens then? The filter has been tricked into “converging” on a false low value. Thereafter, every correct RTT sample (45ms) will produce $\nu >0$ (measurement higher than estimate).

Looking only at the directional gate: $\nu >0$ samples are rejected by the directional update rule — this appears to lock $x\_est$ permanently at 40ms. But KCC has two upward pulling forces:

Drift Detection (Tier-1 / Tier-2)

When $\nu >0$ is rejected, pos_skip_cnt increments. This is the cumulative counter for positive innovations — symmetric to neg_skip_cnt. When pos_skip_cnt reaches the drift threshold (default 16), Tier-1 triggers:

$$corr=\frac{K\cdot |\nu |}{4}$$

i.e., a quarter of the normal Kalman correction is applied (lagging, controlled upward correction). Tier-2 triggers at threshold $16\times 8=128$, applying an even smaller correction ($K\cdot |\nu |/8$).

Drift detection is not as aggressive as Q-boost — it is slow and cumulative, modeling the gradual baseline drift of a physical wireless link or fiber. But once triggered, it moves $x\_est$ gradually upward, back toward the true $T\_prop$.

Q-boost (Large Step Detection)

If $x\_est$ is pulled too far down (e.g., 40ms vs true 45ms, a 5ms gap), and $|\nu |>q\_boost\_thresh$ (≈ 4ms), Q-boost triggers:

• $p\_est$ reset to 1000 (high uncertainty)
• qboost_fired = true, the outlier gate is skipped
• For $\nu >0$, the L12149-L12157 path is taken: $x\_est=x\_est+K\cdot |\nu |$, applying the full Kalman correction, pulling it back in one step

Understanding through a fluid dynamics analogy

RTT sampling in a network is inherently asymmetric:

• Most samples carry $T\_queue>0$ → $\nu >0$ (positive innovations are the norm — a direct consequence of the Lindley recurrence $q\_k+1=\max (0,q\_k+\sum \lambda \_i-C)$: as long as the arrival rate approaches the bottleneck capacity, queues accumulate and RTT rises)
• Sustained $\nu <0$ only appears during PROBE_BW’s DRAIN phase (pacing_gain=0.75x) and after PROBE_RTT drains the queue

Therefore, in the long-term statistical average of the flow, $\nu \ge 0$ samples dominate. neg_skip_cnt handles transient path drops (completed within a few RTTs), while drift + Q-boost handle the persistent upward pull (scales of tens to hundreds of RTTs).

Here is the concrete code verification:

Drift Tier-1 (L12347–L12360): pos_skip_cnt ≥ drift_thresh(16) && jitter_ewma < min_rtt/8 → corr_abs >> 2 (i.e., $K\cdot |\nu |/4$, one quarter of the normal correction). This is a slow response to “sustained small positive biases” — not an immediate correction, but a lagging local correction applied only after accumulation reaches statistical significance.

Drift Tier-2 (L12368–L12382): pos_skip_cnt ≥ drift_thresh × 8 = 128 → corr_abs >> 3 ($K\cdot |\nu |/8$, one eighth). This is a forced correction for “long-term stubborn biases” with an extremely high statistical threshold (Neyman-Pearson upper bound $2^{-128}\approx 2.9\times 10^{-39}$), triggered virtually only when the filter is certain it has drifted away.

Q-boost (L11904–L11911 + L12149–L12157): $\nu >0$, $|\nu |>4$ms and $p\_est\le converged\_val$ → $p\_est$ reset to 1000, applies the full Kalman correction $x\_est=x\_est+K\cdot |\nu |$ in one step.

These three together constitute a persistent upward pull, guaranteeing at the code level that — even if $x\_est$ is incorrectly pulled down by a spurious negative innovation:

3.8 Practical Consequences of an Incorrectly Lowered x_est: Bandwidth Utilization Drop

If $x\_est$ is pulled below the true $T\_prop$ by a false negative, it affects not just filter estimation accuracy — it directly depresses link bandwidth utilization.

KCC defaults to FILTER mode (kcc_rtt_mode = 0, L7705), where model_rtt is taken directly from $x\_est\_us$ (L12534: t_prop_scaled = kcc_rtt_mode == 0 ? ext->x_est : ...). BDP calculation (L10509: target = kcc_bdp(sk, bw, gain, ext)) depends on model_rtt:

$$BDP=bw\times model\_rtt$$

$$target\_cwnd=BDP\times cwnd\_gain(\text{steady-state }cwnd\_gain=2.0)$$

In PROBE_BW steady state, cwnd is limited by target (L10519: cwnd = min(cwnd + acked, target)).

If $x\_est$ is incorrectly pulled to 40ms (true $T\_prop=45$ms):

$$\begin{array}{rl}BDP\_wrong& =bw\times 40\text{ms}\\ BDP\_true& =bw\times 45\text{ms}\\ \text{Utilization loss}& =\frac{45-40}{45}\approx 11\%\end{array}$$

For a 500Mbps link, this translates to a loss of approximately 55Mbps — just for a 5ms $x\_est$ error.

Conversely, if $x\_est$ is overestimated (e.g., min_rtt inflation in BBR mode), BDP is overestimated → cwnd too large → excess injection → packet loss. KCC’s directional gate is precisely designed to prevent this — but the directional gate is asymmetric: it blocks $\nu >0$ updates (preventing $x\_est$ from being inflated by queuing), yet allows $\nu <0$ updates (fast convergence on path drops). The price of this asymmetric design is that $x\_est$ lacks defense against false negatives — and this is exactly the gap filled by the neg_skip_cnt mechanism.

Summarizing this mechanical balance:

• Moving downward: $\nu <0$ passes the directional gate → $x\_est$ is pulled down. neg_skip_cnt ensures this is controlled (3 RTT persistence test), not every isolated noise spike gets through.
• Moving upward: $\nu >0$ is blocked by the directional gate → $x\_est$ cannot rise. But Drift Tier-1/2 and Q-boost provide a controlled, lagged upward channel — they intervene only when statistically significant, preventing the filter from overreacting to noise.
• Cost of error: $x\_est$ too low → utilization loss (insufficient cwnd); $x\_est$ too high → packet loss (excessive cwnd). KCC chooses the asymmetric strategy of “better too low than too high” (directional gate blocks upward moves, neg_skip_cnt ensures the reality of downward moves), because the cost of packet loss far outweighs the cost of temporarily lower utilization.

3.9 Comparison of Multi-Layer Penetration

On the vast majority of links (≥ 500Mbps), the second defense layer (base threshold) directly zeros out TSO artifact penetration. On these links, neg_skip_cnt responds almost exclusively to real path changes.

4. Why Any $\nu \ge 0$ Immediately Resets the Counter

The fundamental premise of the Neyman-Pearson test: “persistent evidence” constituted by N consecutive negatives.

A single sample with $\nu \ge 0$ (even a neutral $\nu =0$) provides evidence in the opposite direction — sufficient to overturn the hypothesis that “we are currently in a continuous downward trend.” Hence:

else {
    ext->neg_skip_cnt = 0;
    ext->last_neg_mstamp = 0;
}

This reset is unconditional — any positive or zero innovation instantly terminates the persistence count. If the path is indeed dropping, subsequent samples will re-accumulate; if it was just random jitter, the reset prevents a false positive.

5. Complete Mathematical Statement

The new code provides the following deterministic guarantee in the negative path-drop scenario:

$$\forall p\_est,Q,R,x\_est\_init,\text{if }{T}_{prop}\text{ drops abruptly from }{t}_0\text{ to }{t}_1<t_0$$

$$\text{and }|t_1-t_0|>dyn\_thresh/S(\text{drop magnitude captured by dynamic threshold})$$

$$\text{and 3 consecutive RTT samples are all near }{t}_1$$

$$\text{then }x\_est\text{ is corrected to }{t}_1\text{ on the 3rd RTT sample}$$

where $S=1024$ is the scaling factor. Convergence time = $3\times t_1$. This promise is completely decoupled from the filter’s internal state.

6. Summary

The old code’s $p\_est\le converged\_val$ gating was a direction-agnostic condition used in both directions — coincidentally valid for positives (“large innovation during convergence = noise”), but state-dependent for negatives (“large innovation during convergence = noise? Depends on the current value of Q”). When Q was suppressed to 0, $p\_est$ could not rebound after a rejection, forming a permanent interlock between the two gates.

The new code uses neg_skip_cnt to provide direction-aware gating for the two directions: negatives go through a persistence test (N=3), positives still use $p\_est$. The Neyman-Pearson sequential framework gives a Type-I error bound of $2^{-3}=12.5\%$, TSO temporal gating filters out burst-compression artifacts, and an immediate reset on $\nu \ge 0$ prevents false accumulation.

Conclusion: In any scenario, convergence from an inflated initial value to the new physical baseline is completed in at most 3 RTTs.