Repository navigation
Core: Add CountNull Aggregation Support - #13981
Conversation
|
@huaxingao PTAL |
|
Could you help review this pr when you have time? @huaxingao Thanks ~ |
|
should we also add to SparkAggregates? https://grep.app/search?f.path=spark%2F&f.repo=apache%2Ficeberg&q=Operation.COUNT_STAR |
I believe CountNull is not supported by Spark because it's not standard SQL, which is why I didn't add it at first; However, it's very useful for data-file statistics, so I think it's reasonable to add the support. |
|
i wonder if we need to update the docs too we have since added CountStar, CountNonNull, and now CountNull |
|
i wonder if this is consider a "spec change" or at least clarification. from https://iceberg.apache.org/spec/#data-file-fields, |
|
In the original java doc |
fc798b1 to
a272efb
Compare
|
Thanks @jackylee-ch for the PR! Thanks @kevinjqliu for the review! |
|
Thanks for your preview. @huaxingao @kevinjqliu |
We’re using
Aggregateto generate per-column statistics from data files—metrics like value counts, min/max, and null counts—but discovered that null counts (CountNull) aren’t currently supported. This PR adds null count support for data files. It also enables any other consumer to easily collect null counts moving forward.