[FeatureRequest] Statistics: Average column width for dynamically-sized types #8274

Hey everyone,

we are currently looking into improving query plans based on available column statistics. Iceberg already supports a number of statistics like null_value or lower_bounds/upper_bounds in data_file struct 1. However, Iceberg spec currently does not support statistics for the average column width (for dynamically sized types like string or binary) in-memory.

We are aware that column_sizes exists but the corresponding map lists the size of columns occupying on disk only. Unfortunately, it is not possible to infer the logical size / in-memory size reliably given the storage size because data files (e.g. Apache Parquet) apply compressing encodings like run-length or dictionary as well as data file compression (e.g. snappy). Hence, we advocate for a new statistics to cover logical / in-memory average width of columns.

Preferably, the addition to the existing Iceberg spec happens as a optional parameter (first) to move quickly and let writer implement the change subsequently. An open PR on Apache Parquet 2 already accommodates this feature request perfectly by adding SizeEstimationStatistics to Parquet's statistics.

Any feedback would be greatly appreciated.