【lwIP】Bug #4 | PCB Held for 38 Minutes After lwip_close() | The Missing FIN_WAIT_1 Timeout

Japanese version available here: 日本語版

This series covers real-world bugs, vulnerabilities, reproduction steps, minimal fixes, and debugging know-how for lwIP — the TCP/IP stack widely used in embedded systems.
▶ English article index: lwIP Troubleshooting Notes

■ Have You Seen These Symptoms?

• You called `lwip_close()`, but Wireshark shows FIN packets being retransmitted for tens of minutes

• After a network failure, new TCP connections suddenly stop working

• You see `memp_malloc` failure logs or PCB pool exhaustion messages

• Even after the network recovers, the device refuses new connections for a while

The root cause may be that lwIP's FIN_WAIT_1 state has no dedicated timeout.

■ Affected Versions and Conditions

• lwIP version: lwIP 1.0.0–2.2.1 (all versions at the time of writing)

• Affected scenario: After calling `lwip_close()`, when the remote peer does not respond to the FIN packet

  • Physical network disconnection

  • Abnormal termination of the remote peer

  • Packet blocking by a firewall (e.g., iptables DROP)

■ What You'll Learn

1. Reproduction steps for PCB being held in FIN_WAIT_1 for up to 38 minutes

   • Setting up a reproduction environment using Linux + Python script

   • The FIN retransmission pattern observable in Wireshark

2. Why it takes 38 minutes (source code analysis)

   • The asymmetric design: FIN_WAIT_2 has a timeout but FIN_WAIT_1 does not

   • How `tcp_backoff[]` causes exponential backoff, stretching retransmission intervals

3. What to consider when choosing a mitigation

   • The difference between retransmission limits, force-close behavior, and state timeouts

   • Side effects to evaluate before applying a change to a product

4. Overview of the patch candidate verified in this article

   • The idea of adding timeout handling for FIN_WAIT_1

   • What to verify in a pcap after applying the change

■ Problem Overview

When you call `lwip_close()`, lwIP sends a FIN packet and transitions the PCB (Protocol Control Block) to the FIN_WAIT_1 state.

TCP connection teardown normally completes in four steps (the 4-way handshake):

• FIN_WAIT_1: Your side has sent a FIN and is waiting for an ACK (acknowledgment) from the peer

• FIN_WAIT_2: The ACK for your FIN has been received; now waiting for the peer's FIN

Under normal conditions, this transition completes within a few seconds. However, if the remote peer does not respond to the FIN at all, the connection stays stuck in FIN_WAIT_1 and keeps retransmitting.

If the remote peer does not respond to the FIN, lwIP keeps retransmitting it up to `TCP_MAXRTX` times (default: 12). With each retransmission, an exponential backoff doubles the waiting time, so reaching the 12th retransmission takes approximately 38 minutes.

FIN_WAIT_2 has a dedicated timeout called `TCP_FIN_WAIT_TIMEOUT` (default: 20 seconds). FIN_WAIT_1 does not, and the PCB is not released until the retransmission limit (`TCP_MAXRTX`) is reached. The root cause is that `tcp_slowtmr()`, the periodic TCP timer function, does not have a dedicated timeout-handling block for FIN_WAIT_1.

Embedded systems typically have only `MEMP_NUM_TCP_PCB` PCBs available (default: 5). A single PCB occupied for 38 minutes reduces the resources available for new connections. If multiple connections enter this state simultaneously, the PCB pool can be exhausted, potentially preventing new TCP connections from being established and affecting overall TCP communication.

■ Reproduction Steps

Environment

• lwIP 2.2.0

• Remote machine: Linux (Ubuntu) + Python 3

Reproduction Mechanism

This setup uses Linux `iptables` to DROP all packets to and from the lwIP device. This creates a controlled version of the situation where a network failure prevents the FIN packet from reaching the peer.

The overall flow is:

1. Start a TCP server script on Linux and wait for the lwIP device to connect

2. The lwIP device connects and exchanges `HELLO`/`OK`

3. After the `OK` response, Linux sets iptables rules to DROP packets to and from the lwIP device

4. After 8 seconds, the lwIP device calls `lwip_close()`

5. Observe the FIN retransmission pattern in Wireshark

Linux Server Script

Run the following Python script (`finwait_server.py`) on Linux. It listens on port 5001 for a connection from the lwIP device, sends an `OK` response, and then automatically sets up iptables DROP.

iptables is the Linux packet filtering mechanism. With `-j DROP`, matching packets are discarded without any response. Here, all packets to and from the lwIP device are dropped, creating a controlled situation where a network failure prevents the FIN packet from being acknowledged. The connection has already been established; only later packets are blocked in both directions.

import socket, subprocess, time, threading

TARGET_IP = "192.168.1.200"  # lwIP device IP address

def run(cmd): subprocess.run(cmd, shell=True)

def drop_packets():
    time.sleep(1)  # Wait 1 second after sending OK
    run(f"sudo iptables -A INPUT  -s {TARGET_IP} -j DROP")
    run(f"sudo iptables -A OUTPUT -d {TARGET_IP} -j DROP")
    print("[Server] iptables DROP active.")

with socket.socket() as srv:
    srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    srv.bind(('', 5001))
    srv.listen(1)
    print("[Server] Waiting for lwIP device connection...")
    conn, addr = srv.accept()
    print(f"[Server] Connected from {addr}")
    data = conn.recv(16)
    print(f"[Server] Received: {data.decode()}")
    conn.send(b"OK\r\n")
    threading.Thread(target=drop_packets, daemon=True).start()
    try:
        elapsed = 0
        while True:
            time.sleep(30)
            elapsed += 30
            print(f"[Server] +{elapsed} sec")
    except KeyboardInterrupt:
        pass
    finally:
        run(f"sudo iptables -D INPUT  -s {TARGET_IP} -j DROP")
        run(f"sudo iptables -D OUTPUT -d {TARGET_IP} -j DROP")
        print("[Server] iptables rules removed.")

sudo python3 finwait_server.py

When you stop it with Ctrl+C, the `finally` block runs and removes the iptables rules automatically.

lwIP Device Reproduction Code

Run the following function as a task. It connects to Linux:5001, exchanges `HELLO`/`OK`, and calls `lwip_close()` 8 seconds later. Adjust `osDelay()` to match the delay API used by your RTOS.

#define FINWAIT_SERVER_IP   "192.168.1.22"
#define FINWAIT_SERVER_PORT 5001
#define FINWAIT_PRE_CLOSE_MS 8000

void tcp_finwait_task(void *arg)
{
    int sock = lwip_socket(AF_INET, SOCK_STREAM, 0);
    struct sockaddr_in addr = {
        .sin_family = AF_INET,
        .sin_port   = PP_HTONS(FINWAIT_SERVER_PORT),
    };
    ip4addr_aton(FINWAIT_SERVER_IP, (ip4_addr_t *)&addr.sin_addr);

    lwip_connect(sock, (struct sockaddr *)&addr, sizeof(addr));
    lwip_send(sock, "HELLO\r\n", 7, 0);

    char buf[16] = {0};
    lwip_recv(sock, buf, sizeof(buf)-1, 0);  /* Receive "OK" */

    osDelay(FINWAIT_PRE_CLOSE_MS);
    lwip_close(sock);  /* ← Transitions to FIN_WAIT_1 */

    /* Display elapsed time while waiting for PCB release */
    int elapsed = 0;
    while (1) {
        osDelay(30000);
        elapsed += 30;
        printf("[FINWait] +%d sec elapsed since lwip_close()\n", elapsed);
    }
}

Observe FIN Retransmission in Wireshark

Set a capture filter to show only TCP packets from the lwIP device's IP address:

ip.addr == 192.168.1.200 && tcp

Observe the Behavior

When the lwIP device starts, the following sequence occurs:

• It connects to Linux:5001

• Sends `HELLO` → Linux responds with `OK` → iptables DROP activates

• 8 seconds later, `lwip_close()` is called → FIN is sent → transitions to FIN_WAIT_1

• FIN packets appear in Wireshark, retransmitting with exponentially increasing intervals

The following is the Wireshark result observed on actual hardware. From FIN to the final Retx 12 (at 2308.743 s), 38.3 minutes elapsed.

Pkt  8: t=   8.036s  FIN (initial)
Pkt  9: t=  10.695s  Retx 1    Δ=  2.659s
Pkt 10: t=  17.151s  Retx 2    Δ=  6.456s
Pkt 11: t=  28.695s  Retx 3    Δ= 11.545s
Pkt 12: t=  52.695s  Retx 4    Δ= 24.000s
Pkt 13: t= 100.696s  Retx 5    Δ= 48.000s
Pkt 14: t= 196.696s  Retx 6    Δ= 96.000s
Pkt 15: t= 389.525s  Retx 7    Δ=192.829s
Pkt 16: t= 772.698s  Retx 8    Δ=383.173s  ← Plateau starts at 384 s
Pkt 17: t=1156.719s  Retx 9    Δ=384.021s
Pkt 18: t=1540.701s  Retx 10   Δ=383.982s
Pkt 19: t=1924.735s  Retx 11   Δ=384.034s
Pkt 20: t=2308.743s  Retx 12   Δ=384.008s  ← Final retransmission
(No further packets. No RST. Final retransmission observed after about 38.3 minutes.)

FIN transmission to final retransmission: 2300.7 s (approx. 38.3 minutes). After that, `tcp_slowtmr()` detects that the retransmission limit has been reached and releases the PCB according to the source code path.

Alternative Reproduction Method (Physical Cable Disconnection)

In environments where iptables is unavailable, the same behavior can be reproduced by physically unplugging the LAN cable immediately after the `HELLO`/`OK` exchange. The difference from iptables DROP is that packets sent by the remote peer also stop arriving after disconnection. From the perspective that FIN responses no longer reach the device, a similar situation can be created.

■ Reproduction, Root Cause, and Fix

This section has covered the affected conditions, problem overview, and reproduction steps.
Next, as an introduction to the source code analysis, we briefly cover the minimum needed to understand the overall picture of the problem.
Detailed code tracing, fix methods, verification after applying the fix, and official status are covered in the paid section below.

For readers dealing with the same issue, or manufacturers who need to patch already-shipped products, the paid section provides concrete, immediately usable information.

For an overview of lwIP, per-vendor SDK differences, and why lwIP bug information matters, see the introductory article:

【lwIP】Common Bugs, Vulnerabilities and Fixes | Embedded Engineer's Field Notes

■ Source Code Analysis

The following analysis focuses on `src/core/tcp.c` in lwIP 2.2.0.

[Analysis 1] FIN_WAIT_2 Has a Timeout; FIN_WAIT_1 Does Not

`tcp_slowtmr()` is the slow timer function called every 500 ms. It processes timeout checks for each PCB in sequence.

The FIN_WAIT_2 timeout check is implemented inside `tcp_slowtmr()` in `tcp.c`:

/* Check if this PCB has stayed too long in FIN-WAIT-2 */
if (pcb->state == FIN_WAIT_2) {
  if (pcb->flags & TF_RXCLOSED) {
    if ((u32_t)(tcp_ticks - pcb->tmr) >
        TCP_FIN_WAIT_TIMEOUT / TCP_SLOW_INTERVAL) {
      ++pcb_remove;
      LWIP_DEBUGF(TCP_DEBUG,
        ("tcp_slowtmr: removing pcb stuck in FIN-WAIT-2\n"));
    }
  }
}

`TCP_FIN_WAIT_TIMEOUT` is defined in `tcp_priv.h` with a default value of 20000 ms (20 seconds) in 2.2.0. Dividing by `TCP_SLOW_INTERVAL` (500 ms) gives 40 ticks = 20 seconds.

There is no equivalent block for FIN_WAIT_1. When `tcp_slowtmr()` processes a FIN_WAIT_1 PCB, it performs no timeout check — only the retransmission logic continues running.

[Analysis 2] `tcp_backoff[]` and RTO Calculation: Why 38 Minutes?

The retransmission interval is determined by the following array defined in `tcp.c`: