Retrieve One Row from a Table, Not the Whole Table: Row-Level Chunks for RAG

A car-insurance contract prints its guarantees as a 40-row table: one row per covered event, with columns for the event, its cap, its deductible, and the eligibility condition. A user opens the search bar and asks “what is the cap for vehicle theft?”. The naive retrieval move indexes the whole table as one chunk. When the question matches, the generation model receives 39 rows of unrelated events on top of the one that answers, and has to guess which line the reader meant. Sending the whole table makes the model do the filtering the retriever should have done, and the unit that actually matches the question is one row, not the whole rectangle.

This article is part of Part II of Enterprise Document Intelligence, a series that builds an enterprise RAG system from four bricks. The retrieval brick treats retrieval as filtering rather than vector search, runs keyword and embedding signals in parallel, lets an LLM arbiter rank the finalists, and routes long documents through their table of contents. This companion handles the case they leave open: a document whose answer lives inside a table, where the unit that fits the question is one row.

🧭 New to the series? Start with the map: Prompt, Context, Loop sets out the three engineering layers every RAG system is built on, the prompt (the call itself), the context (what fills the model’s window), the loop (when the next call fires and when it stops), and walks the whole series through that lens, article by article. It is the shortest way to see what is covered and where this one sits.

*Article 7sexies sits in Part II as a retrieval companion. – Image by author*

📓 The runnable companion is on GitHub: load Table 1 of the Attention paper, watch serialize_table_rows turn it into four row-level chunks, then run a targeted keyword query and see it return the one row instead of the whole table. On GitHub: doc-intel/notebooks-vol1.

*The public companion-code repo at doc-intel/notebooks-vol1 – Image by author*

We work through this on Attention Is All You Need (Vaswani et al. 2017; arXiv non-exclusive distribution license, declared on the arXiv abstract page). Its Table 1 (page 6) is a compact real-world case: four rows, four columns, every row is a different answer. Runnable code paths call OpenAI services governed by OpenAI’s Terms of Use. Parsing uses Docling (MIT license).

1. Table vs row: the unit of retrieval

A table on paper is a bounded region: a rectangle of cells with a header row on top and body rows below. A retrieval system that treats the whole rectangle as one chunk collapses that structure. The chunk either matches (and the generation model receives every row, most of them irrelevant to the question) or it does not (and the generation model receives nothing, because the one relevant row is buried under all the other rows the scorer had to average over).

The mismatch is between the document’s unit of information (one rectangle) and the reader’s unit of question (one row). A question like “cap for vehicle theft?” asks about one row. A question like “which events are covered?” asks about the whole rectangle. Both are legitimate. A retrieval brick that only offers the rectangle answers the second question and mishandles the first.

The fix is not to pick one scale over the other, but to make both scales available and let the dispatcher pick. That is what this article builds, on top of the parsing brick’s existing output.

2. Building the row-level index

Two steps turn a parsed table into a retrievable index: read the shape the parser emits, then serialize each body row into its own chunk.

2.1 What the parser already emits

Docling and Azure Document Intelligence both emit tables as markdown-pipe lines inside line_df. A table with 4 body rows becomes 6 lines: a header row (| Col1 | Col2 | ... |), a separator row (| --- | --- | ... |), and one body row per data row, in reading order. No extra _type column survives the public line_df contract. The pipe pattern in text is the signal.

*The pipe format is the parser’s contract for a table, and what the row-level serializer reads. – Image by author*

The rules of the pipe format are the same across docling and Azure Layout, and light enough to detect without a per-parser branch:

A pipe row starts and ends with |, with at least one interior pipe.
The separator row (interior of dashes, colons and pipes) marks the split between the header and the body.
Consecutive pipe rows on the same page, or across an immediate page break, belong to the same table.
Anything else, including a non-pipe line in between, resets the group.

The four rules above are the whole contract. The row-level primitive below reads only line_df and needs nothing else from the parser.

2.2 One chunk per table row

The serialize_table_rows function turns each body row of every table into a retrievable chunk. A pandas.DataFrame in, a pandas.DataFrame out, the same shape Article 7B’s retriever uses, so the two bricks compose over the one line_df. The whole function is a small scan: group the pipe lines into tables, read the header off the row above the separator, and emit one output row per body line.

def serialize_table_rows(line_df: pd.DataFrame) -> pd.DataFrame:
    """One retrievable chunk per body row of every table."""
    lines = line_df.sort_values(["page_num", "line_num"])
    tables = group_contiguous_pipe_rows(lines)   # runs of | ... | lines

    out = []
    for tid, table in enumerate(tables, start=1):
        sep = first_separator_row(table)         # the | --- | --- | line
        headers = split_cells(table[sep - 1])    # the row above the separator
        headers, body = fold_multirow_header(headers, table[sep + 1:])
        for row_idx, line in enumerate(body):
            cells = split_cells(line.text)
            out.append({
                "table_id": f"t{tid}",
                "page_num": line.page_num, "line_num": line.line_num,
                "column_headers": headers, "row_cells": cells,
                # headers travel with the values, so a match reads like a sentence
                "row_serialized":
                    " | ".join(f"{h}: {c}" for h, c in zip(headers, cells)),
            })
    return pd.DataFrame(out)

The function returns a new frame, not a mutation of line_df. The parsing brick’s frame stays the point of truth for the document’s geometry (bbox, position, page); the row-level frame is a second index on top of it. Downstream retrievers that never enable row-level indexing keep working unchanged. The ones that do read the second frame explicitly, so nobody has to check a chunk_type tag.

The serialization format. Each body row becomes col: val | col: val | .... The choice is deliberate: an LLM reads that shape as naturally as a paragraph, and the retriever that keyword-matches or embeds paragraphs today can index row_serialized values with no change. The reader who receives a match sees a self-explaining line (the column names travel with the values) instead of a naked cell.

Column-value pairs also compose with the standard citation contract from Article 8. When the generation model quotes a specific value, the quote lands on the body row’s (page_num, line_num), exactly like any other span.

3. Retrieving at two scales

The row-level index is a second scale, not a replacement. The dispatcher picks the scale from the question, and two worked examples (the guarantees table from the intro, then a real published paper) show the payoff.

3.1 Two scales, one dispatcher

Three question shapes cover most of what users ask a table, and each maps to a scale.

Targeted lookup (“cap for vehicle theft?”, “BLEU for the Transformer big model?”, “salary of the deputy CFO?”). Route to the row-level index. Return one row plus its column headers. Generation receives exactly what it needs.
Synthesis (“which events are covered?”, “what layer types does the paper compare?”, “list every executive on the compensation grid?”). Row-level still fires, but every row of the same table matches. The dispatcher notices the shared table_id and widens back to the whole table before generation.
Mixed (“summarise the vehicle-theft cap and compare it with the fire cap”). Two rows on the same table, generation stitches the answer from both. The row-level index makes this cheap; a whole-table dump would have worked too, but at a higher token cost.

*Row granularity is a second scale, not a replacement, chosen by the dispatcher from question shape. – Image by author*

The widening rule from row-level to whole-table is worth naming: when k body rows of the same table_id all match the question, and k covers most of the table, treat the match as table-level. In practice, a threshold like k / n_body_rows ≥ 0.6 catches synthesis queries. Below that threshold, the row-level answer is the honest one. A synthesis pretending to summarise on the strength of 2 out of 40 rows would be a hallucination the arbiter of Article 7C should already refuse.

3.2 The guarantees table from the intro

Back to the car-insurance contract that opened this article. Its guarantees table has one row per covered event and four columns: the event, its cap, its deductible, and the eligibility condition. Here is a slice of it (the full contract runs to about forty rows).

*The guarantees table from the intro, a slice of the full contract. – Image by author*

The question from the intro, “what is the cap for vehicle theft?”, is a targeted lookup. Serialize the table, then keyword-match on the serialized rows.

row_df = serialize_table_rows(line_df)          # one chunk per body row
hit = row_df[row_df["row_serialized"].str.contains("Vehicle theft")]
# 1 row, 122 chars:
#   "Covered event: Vehicle theft | Cap: 25,000 EUR |
#    Deductible: 500 EUR | Eligibility condition: Tracker installed and active"

The match returns one row of 122 characters: the Vehicle theft event with its cap, deductible and condition, nothing else. The whole eight-row slice is 943 characters, a 7.7× ratio already. On the full forty-row contract the same single-row answer is roughly a 40× saving, since the ratio is just the row count. The generation model reads the one line it needs, so it cannot answer with the fire cap or the vandalism deductible by mistake.

3.3 The same on a real PDF: the Attention paper

The guarantees table is hand-built to match the intro; here is the same mechanism on a real published PDF. The Attention paper has four tables. Table 1 on page 6 is the compact case that shows the mechanism end to end. It has four body rows, one per layer type: Self-Attention, Recurrent, Convolutional, Self-Attention (restricted). The columns are Layer Type, Complexity per Layer, Sequential Operations, Maximum Path Length. Here it is as printed in the paper:

The actual Table 1 the parser reads, straight from the PDF. – Image by author (crop of arXiv:1706.03762v7, page 6, under the arXiv non-exclusive distribution license)

Two questions test the two scales.

*Every layer type is a distinct chunk; Self-Attention is the accent line. – Image by author*

Targeted query. “What is the complexity of self-attention per layer?”. Keyword-match on row_serialized returns the Self-Attention row (page 6, line 4), 124 characters of context. A looser match also surfaces the Self-Attention (restricted) variant on line 7; either way the retriever hands back the one or two rows that match, not the whole rectangle. The whole Table 1 is four rows totalling 528 characters, a 4.3× ratio for the single-row hit on this compact four-row table. The gap tracks the row count, the same way it did on the guarantees table in the previous section.

Synthesis query. “What layer types are compared?”. Keyword-match on Layer Type: returns four hits, all on table_id = t1. Every row of the table matches, so the dispatcher widens back to the whole table. Same outcome as a table-level retriever, no regression.

*Context-size ratio measured on Attention Table 1; the gap scales with the number of rows. – Image by author*

4. Edge cases and pipeline wiring

Two practical concerns remain: the one table shape the serializer has to handle itself, and how the row-level index plugs into the existing retriever.

4.1 The hard case: multi-row headers

Real parsers do not always produce a single header row. On the Attention paper Table 2 (page 8, BLEU and Training Cost), the printed header spans two lines: Model / BLEU / Training Cost on top, then EN-DE / EN-FR under both BLEU and Training Cost. Docling flattens that into a first pipe row with blank cells (Model, BLEU, , `Training Cost (FLOPs)`,), a separator, then a body row that is really the second header line, before the numbers begin.

A single-line serializer reads the flattened first row as the header and the second header line as the first body row. A keyword search for EN-DE then hits that label-only row before the numeric rows below it. This is the one table shape the primitive has to handle itself, because it is common and the failure is silent.

The fix stays inside the serializer, in the fold_multirow_header step the core loop already calls. The signal is the blank cell: a header cell is empty only when the parser flattened a cell that spanned several columns. When that happens, forward-fill the spanned label rightward, then check whether the first body row is really a sub-header (all labels, no digits, with real numbers on the row below). If it is, concatenate the two header levels column by column and drop the sub-header line from the body.

def fold_multirow_header(headers, body):
    """Fold a spanned two-line header into one labelled header row."""
    if "" not in headers or len(body) < 2:   # nothing to fold
        return headers, body

    headers = forward_fill(headers)      # a merged cell blanks the columns it spans
    subline = split_cells(body[0].text)
    is_subheader = (
        len(subline) == len(headers)
        and not any(has_digit(c) for c in subline)               # all labels
        and any(has_digit(c) for c in split_cells(body[1].text)) # numbers below
    )
    if is_subheader:
        headers = [f"{t} {s}".strip() for t, s in zip(headers, subline)]
        body = body[1:]                  # the sub-header line is not a data row
    return headers, body

On the real docling parse of Table 2, this turns the header into Model, BLEU EN-DE, BLEU EN-FR, Training Cost (FLOPs) EN-DE, Training Cost (FLOPs) EN-FR, drops the label-only line, and serializes the numeric rows against the merged labels. A search for EN-DE now lands on the data rows, where it belongs.

The guard is what makes this safe to run on every table. It fires only when the header carries a blank cell and the suspected sub-header is all text and the row beneath it carries digits. An ordinary single-line-header table, or an all-text glossary table, keeps every body row untouched. Upstream is still the better place to normalise multi-row headers, since a parser that emits one logical header row fixes it for every consumer, not just retrieval. Until that lands, the serializer no longer breaks silently on the shape.

4.2 Where it plugs into the pipeline

The row-level frame is the parallel index from section 2.2, wired into the retriever by a single activation.

row_df = serialize_table_rows(line_df) is computed once and cached next to line_df (output/<doc-stem>/parsing/row_df.parquet).
The retriever gains a use_row_level: bool flag alongside use_toc, use_keywords, use_dense. On, the row-level frame is indexed with the same keyword and embedding primitives that already run on paragraphs.
The question parser (Article 6) sets the flag by question shape: targeted_lookup on a document with tables turns it on, synthesis leaves it off, mixed turns it on and lets the widening rule from section 3.1 do the fusion.

Nothing about the paragraph-level retriever changes.

Summary

Tables in a document carry structured information that a paragraph-level retriever averages away. This article added a second retrieval scale on top of the existing line_df contract: serialize_table_rows(line_df) → row_df turns each body row of every detected table into a chunk paired with its column headers, keyed on (page_num, line_num) for citation. The dispatcher picks the scale by question shape. Targeted queries route to row-level, synthesis queries widen back to the whole table, mixed queries stitch two or three rows. On a slice of the insurance guarantees table a targeted lookup returns one 122-character row where the whole slice is 943, and on the Attention paper Table 1 the same move sends 124 characters against 528; the ratio is about the row count, so a full contract saves far more. Multi-row headers, the one shape a single-line serializer gets wrong, are folded inside the serializer: a spanned header is forward-filled and its sub-header line concatenated into one labelled row, guarded so an ordinary table is never touched. The row index sits alongside line_df, not inside it. The parsing brick’s contract is untouched, and every existing consumer of line_df keeps working unchanged.

Sources and further reading

Docling documentation, table structure preservation: docling-project/docling.
Vaswani, A. et al. Attention Is All You Need. arXiv:1706.03762, 2017. arxiv.org/abs/1706.03762. Table 1 (page 6, layer complexity) and Table 2 (page 8, BLEU and Training Cost) are the worked examples of this article.
serialize_table_rows, the row-level serializer, with its test suite, ships in the companion notebooks repository (doc-intel/notebooks-vol1).
Reproducible probe: scripts/dev/probe_row_level_retrieval.py regenerates the cached row_df_docling.parquet used in the worked example and the multi-row-header fix.

Earlier in the series (the ones still earning, worth the click):

Retrieval