S3 Reference

Options and behavior of the hardwood-s3 module; Reading from S3 shows how to use it.

S3Source builder options

Option	Default	Effect
region(String)	—	AWS region. Required unless endpoint is set.
endpoint(String)	—	Custom endpoint for S3-compatible services (R2, GCS HMAC, MinIO). Region may be omitted when set.
pathStyle(boolean)	false	Path-style addressing, for services that require it (MinIO, SeaweedFS).
credentials(...)	—	Static S3Credentials, an S3CredentialsProvider, or the AWS chain via hardwood-aws-auth. Required.
connectTimeout(Duration)	10s	Maximum time to establish a TCP connection. Ignored when a custom httpClient is supplied.
requestTimeout(Duration)	30s	Maximum time for an individual HTTP request to complete.
maxRetries(int)	3	GET retries on transient failures (HTTP 500, 503, network errors) with exponential backoff and jitter. 0 disables retries.
httpClient(HttpClient)	internal	Caller-supplied client for full transport control. The caller is responsible for closing it; S3Source.close() will not.
rangeBacking(RangeBacking)	RangeBacking.NONE	Whether files opened from this source cache the byte ranges they fetch. See Range backing.
tempDir(Path)	java.io.tmpdir	Directory holding the sparse backing file under RangeBacking.SPARSE_TEMPFILE. Ignored under RangeBacking.NONE.

When the retry budget is exhausted, the read fails with an IOException. For an HTTP error status the message has the form Failed to open s3://<bucket>/<key>: HTTP <status> <body> when opening the file and Failed to read range [<start>, <end>) from s3://<bucket>/<key>: HTTP <status> <body> for a range read; for network errors the original IOException message is preserved.

A range read must be answered with HTTP 206. An endpoint that ignores the Range header and answers HTTP 200 with the whole object makes the read fail with an IOException whose message has the form Failed to read range [<start>, <end>) from s3://<bucket>/<key>: HTTP 200, expected 206; the endpoint ignored the Range header.

Footer pre-fetch

open() issues one GET for the last 64 KB of the object, or the whole object if it is smaller. That single round-trip discovers the object size, so there is never a separate HEAD request. Those bytes are kept for as long as the file is open, and any read falling entirely inside that trailing window is answered without a further request. An HTTP 200 answer to this GET is accepted: the whole object then forms the window, and no read goes to the network again.

For most files the window also carries the Parquet footer, which is why opening a file usually costs one GET. It is a cost heuristic rather than a guarantee: footer size grows with row groups × columns and with the size of the per-column statistics, so a wide schema, a heavily split file, or long string min/max values can push the footer past 64 KB. Reading it then costs one extra GET. When the context's MetadataSource supplies the footer, the open reads only the 8-byte trailer, which lies in the window, so that extra GET is not made.

The ETag header of this response is the object's InputFile.identity(), quotes included. It is empty when the endpoint sends no ETag.

This applies to every S3InputFile, independent of the range backing below.

I/O behavior

Hardwood coalesces the column chunk reads within each row group into as few HTTP requests as possible. The page index of the columns a read uses is fetched for several row groups at a time, one request for the offset indexes and, on a filtered read, one for the column indexes and one for the bloom filters it checks; a filtered read may add one request per dictionary it checks. Column projection, page-level predicate pushdown, and head(n) on an unfiltered read narrow the byte ranges before coalescing.

Without a row limit, the reader fetches all projected (and filtered) column data for an entire row group when any column enters it, even if the consumer stops reading early. head(n) on the buildRowReader() builder truncates each column's fetch to the pages covering the needed rows. With a filter, head(n) counts matching rows, which are not known before the pages are decoded, so it does not narrow the fetch.

Range backing

RangeBacking sets whether an S3InputFile also keeps the bytes it fetches outside that window:

Value	Behaviour
NONE (default)	Nothing beyond the footer pre-fetch is kept. Every other non-empty readRange issues an HTTP GET, including a repeat read of a range already fetched.
SPARSE_TEMPFILE	Fetched ranges are written into a sparse temp file mmapped into the process. A read covered by already-fetched bytes is served from the mapping with no HTTP GET; a read spanning a gap fetches only the missing sub-ranges.

The value changes how many HTTP requests go out; both give the reader identical bytes. The cache is per S3InputFile and lives until that file is closed.

SPARSE_TEMPFILE applies to a whole S3Source, so it suits workloads that re-read the same byte ranges; hardwood dive opts in for every file it opens. Its properties:

• Disk and memory footprint scale with bytes fetched, not with file size, on filesystems that support sparse files (ext4, xfs, apfs, ntfs).
• Files larger than Integer.MAX_VALUE bytes (2 GB) cannot be range-backed; open() throws an UnsupportedOperationException naming the limit. Read those with RangeBacking.NONE.
• The backing file is created in tempDir, which must exist and be writeable with room for the worst-case bytes fetched per file. S3Source.builder().build() rejects a directory that is missing or not writeable.
• close() deletes the backing file, but the mapping behind it is released by the garbage collector, so the address space outlives the close() call. On Windows, where a mapped file cannot be deleted, the file itself also outlives it, until JVM exit.

Fetch-cost counters

Each S3InputFile tracks the network I/O it has performed since open():

• networkRequestCount() — HTTP GET requests issued: the footer pre-fetch plus every column-chunk range fetch. Reads answered from the pre-fetched footer window are not counted, nor are reads answered from the cache under RangeBacking.SPARSE_TEMPFILE.
• networkBytesFetched() — total bytes fetched across those requests, counted the same way.

Read the counters before the try-with-resources block exits, while the S3InputFile is still open.

Not supported

• Anonymous (unsigned) requests — S3Credentials requires an access key + secret key pair; there is no built-in mode for public buckets without credentials.
• Requester-pays buckets — Hardwood does not send the x-amz-request-payer header. Reads against requester-pays buckets will fail with 403 AccessDenied even with valid credentials.