Repository navigation
[Docs] Add Spark MERGE INTO fields in snapshot summary - #15390
Conversation
github-actions
Bot
added
the
Specification
|
cc @singhpk234 @amogh-jahagirdar sorry i didnt doc it until now. Btw, it is a trivial change, I think we dont need spec vote? |
|
actually not sure it belongs in iceberg core spec, but its suggestion in : #15014 (comment) |
singhpk234
left a comment
singhpk234
left a comment
There was a problem hiding this comment.
LGTM too, thanks @szehon-ho
my understanding is same too, lets see if folks have opinion otherwise :)
|
Since these are Spark-specific fields, it might fit better in docs/docs/spark-writes.md under the ### MERGE INTO section? |
|
yea, actually when i was putting the section in spec.md, it seemed a bit off as its engine-specific, so I put in in spark-writes as suggest as @huaxingao |
|
@szehon-ho Please update the PR title and description. |
amogh-jahagirdar
changed the title
amogh-jahagirdar
left a comment
amogh-jahagirdar
left a comment
There was a problem hiding this comment.
Minor comment on including the version , other than that looks great. Thank you @szehon-ho
|
|
||
| #### Snapshot summary | ||
|
|
||
| After a `MERGE INTO` commit, the [snapshot summary](../../spec.md#optional-snapshot-summary-fields) may include the following fields. Each value is the string form of a non-negative count. A field is omitted when the value is unknown (e.g., not reported by Spark). |
There was a problem hiding this comment.
These new summary fields are produced after Spark 4.1+ right? Should we call that out specifically?
There was a problem hiding this comment.
Good catch, done! Sorry for delay
kevinjqliu
left a comment
kevinjqliu
left a comment
There was a problem hiding this comment.
LGTM rendered locally
|
Thanks @kevinjqliu for checking, @singhpk234 @amogh-jahagirdar @huaxingao @manuzhang for reviews! |
6 participants
Add the snapshot properties added in : #15014 to the Spark docs.