Row Selection¶
Hardwood gives a RowReader five controls for
narrowing what comes back: filter, head, tail, skip, and the byteRange row-group
predicate. Row selection counts over the result set, not over the file.
Two questions: which rows, and where they live¶
Selection has two independent axes, and each control belongs to exactly one of them.
- Which rows you want: the logical result. A
FilterPredicate(WHERE), and the positional controlshead,tail, andskip. - Where data lives in the file: the physical layout.
RowGroupPredicate.byteRange(...), at row-group granularity. This is the lever for splitting a file across parallel readers.
Keeping these apart is what makes the controls predictable: physical positioning is byteRange's
job, and nothing else's.
The result set¶
The FilterPredicate defines a result set: the rows that match, in file order. The
positional controls count over that result set, exactly like SQL clauses count over the rows that
survive a WHERE:
| Control | SQL analogue | Meaning over the result set |
|---|---|---|
filter |
WHERE |
the rows that match the predicate, in file order |
head(n) |
LIMIT n |
the first n matching rows |
skip(n) |
OFFSET n |
discard the first n matching rows, keep the rest |
So skip(n).head(k) is OFFSET n LIMIT k: it returns at most k matching rows, starting after
the first n matches. None of these count physical rows, so a matching row sitting deep in the
file is the first row head(1) returns. (tail is not in this table because it cannot
combine with a filter; see Supported combinations.)
The no-filter coincidence¶
With no filter, every row matches, so the result set is the whole file and a logical
position coincides with a physical one. That is the case where skip(n) can be a true seek:
it begins at physical row n, and earlier row groups are never opened. It is an O(1-row-group) jump
that fetches none of the bytes in between. head(n) is the first n rows of the file.
Add a filter and the coincidence breaks. Row-group statistics bound the values in a group, not
the count of rows that match, so the reader cannot know how many matches lie ahead without
looking. head and skip then stream over the matched rows, decoding earlier groups to count
them, though groups whose statistics prove no row can match are skipped wholesale.
Physical splitting is separate¶
byteRange(start, end) assigns each row group to exactly one of a set of disjoint byte ranges
(see Split-Aware Reading). It composes with the
logical controls by intersection: under byteRange(...).filter(p).skip(n).head(k), the result
set is the matching rows within this reader's row groups, and skip/head count over that.
Supported combinations¶
head, skip, and byteRange each compose with a FilterPredicate. tail + filter is rejected
at build(): the last n matching rows have no row-group-statistics shortcut, so it would
require a reverse scan over the whole file.
Further reading¶
- Predicate Pushdown, Projection, Limits, and Splits — the step-by-step guide for each control.
- How a Parquet File Is Laid Out — row groups, pages, and why statistics let whole groups be skipped.