Iceberg Doctor

Forensics and visualization toolchain for Apache Iceberg tables. Patterned after Java Flight Recorder: a collector reads Iceberg metadata and produces a single self-contained artifact file (.icediag.json); a separate viewer renders that artifact through multiple lenses.

Iteration E + dedup — collector reads the full chain (root metadata.json → manifest-list avros → manifest avros), computes per-data-file read amplification, and emits a deduped 1.1 artifact.

What's populated:

• table, snapshots, schemas, partition_specs, sort_orders

• manifest_lists[remote_path] = { size_bytes, snapshot_id, parent_snapshot_id, sequence_number, manifest_paths: [string, …] } — just paths, not inline summaries.

• manifests[remote_path] = { size_bytes, content, partition_spec_id, entries: [...], summary: {...} } — the per-manifest summary (manifest_length, sequence numbers, file/row counts, partitions) lives here once. Each distinct manifest is read exactly once.

• files: per-current-snapshot file index keyed by file_path. Each data file has a read_amplification block: { factor, position_delete_count, equality_delete_count }. The full applicable-deletes path lists are off by default (--include-amplification-detail to include them).

• findings: detection-rule output. Each finding has code, severity (high/medium/low/info), title, description, evidence, and suggested_action. Rules live under src/iceberg_doctor/findings/ — adding a rule is one new file plus a registration in findings/__init__.py.

Initial detection rules (run automatically on every collect):

• manifest_bloat — current snapshot has too many manifests.
• single_file_manifest_ratio — high % of manifests hold a single file (streaming-write fingerprint).
• empty_manifest_skeletons — manifests where every entry is DELETED.
• delete_accumulation — delete-files-to-data-files ratio is high.
• high_amplification — p95 read amplification is high; lists worst offenders.
• unpartitioned_with_deletes — table is single-partition and accumulating position deletes (worst-case layout).
• high_broadcast_delete — surfaces delete files that apply to many data files (the inverse of read amplification — these are the highest-impact deletes to compact first).
• compaction_staleness — flags long stretches of snapshots without a replace operation (no maintenance run).

Empty for now: stats.

Manifest avros and manifest-list avros are resolved by basename next to the metadata.json. Missing files emit warnings, not hard errors.

Flags:

• --no-include-manifests skips manifest-avro reading; no files index or amplification, but per-manifest summaries are still recorded.
• --include-column-stats adds per-file column lower/upper bounds and null counts. Substantially larger output; needed by the future query-simulator lens.
• --include-amplification-detail adds the full applicable-deletes path lists to each data file's amplification block. Off by default — these lists dominated artifact size before dedup.

Format version: 1.1. The viewer reads from this format; 1.0 artifacts won't render correctly.

• Python 3.13+
• uv for environment and dependency management

uv sync

This installs the package in editable mode and sets up dev dependencies (pytest).

# Produce an artifact
uv run iceberg-doctor collect <path-to-metadata.json> -o <output.icediag.json>

# Print the findings from an artifact (color-coded by severity)
uv run iceberg-doctor findings <output.icediag.json>

Omit -o to write the artifact to stdout.

A real sample table's metadata directory ships under SampleMetadata/metadata/. The root metadata file is 05292-197451da-ae69-43f2-82f8-ac59104ba04c.metadata.json.

uv run iceberg-doctor collect \
  SampleMetadata/metadata/05292-197451da-ae69-43f2-82f8-ac59104ba04c.metadata.json \
  -o report.icediag.json

Inspect the result with jq:

jq '.table'                           report.icediag.json
jq '.snapshots | length'              report.icediag.json
jq '.snapshots[0]'                    report.icediag.json
jq '.table.refs'                      report.icediag.json

For the bundled sample you should see 107 snapshots, 1 schema, 1 partition spec, and a single main branch ref.

A static, zero-build web viewer that renders the snapshot DAG from a .icediag.json artifact. No npm install, no dev server required.

# from the repo root
python3 -m http.server 8000
# then open http://localhost:8000/viewer/ and drag a .icediag.json onto the page

(You can also open viewer/index.html directly — the CDN scripts work over file://.)

What you'll see:

• Snapshot DAG laid out left-to-right (oldest → newest), parent edges as arrows.
• Nodes colored by operation: append, overwrite, delete, replace, other.
• Node size scales with total-records (log-scaled).
• The current snapshot has a white border.
• Each table.refs entry (e.g. main) renders as a labeled pill linked to its snapshot.
• Hover (or tap) a snapshot for a tooltip with id, parent, sequence number, timestamp, full record/file/byte totals, additions in this snapshot, and delete/data ratios.
• Click a snapshot to render its manifests in the bottom pane: green = data manifests, red = delete manifests, sized by total file count. Hover a manifest for quick stats; click a manifest to open a sticky right-side detail panel with file counts, size distribution, top files, and per-data-manifest amplification rollup.
• A Findings button in the top bar shows detected problems for the loaded artifact, ranked by severity, each with a suggested remediation action. Click any finding to jump the DAG to the relevant snapshot (the manifest sub-graph and rollup re-render automatically).
• A Charts button opens a full-viewport overlay with charts across snapshot history: a stacked-area for file content composition (data / position deletes / equality deletes), data file count, delete file count, total bytes, total records, manifest count (data vs delete), and read-amplification p50 / p95 / max.
• Click the table name in the top bar to open a side panel with table identity, refs (with retention policies), table properties, the active schema, and the partition spec / sort order.

uv run pytest

The smoke test runs the collector against the bundled sample and asserts the artifact's shape.

src/iceberg_doctor/
  artifact.py             # v1.0 artifact shape + constants
  collect.py              # orchestrates the full read; builds the files index
  manifest_list_reader.py # parses snap-*.avro
  manifest_reader.py      # parses *-m*.avro entries
  amplification.py        # computes read_amplification per data file
  findings/               # detection rules; one file per rule
  cli.py                  # `iceberg-doctor` entry point
tests/
  test_collect.py
viewer/
  index.html     # static viewer (drag-drop a .icediag.json)
  app.js
  styles.css
SampleMetadata/   # bundled sample table metadata for development