Is your feature request related to a problem? Please describe.
As a user of the DynamoDB source using a custom document_id or document_version from my DynamoDB items, I am not able to keep deletions on my DDB table in sync with OpenSearch, because Data Prepper only reads the newImage of DDB stream records, and does not write the old image
Describe the solution you'd like
A new parameter under the stream block of DDB source
source:
dynamodb:
stream:
use_old_image_for_deletes: true
When this is set to true, Data Prepper would check stream records for the REMOVE action, and then it would construct the Event from the old image, rather than the new empty image.
Describe alternatives you've considered (Optional)
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
Is your feature request related to a problem? Please describe.
As a user of the DynamoDB source using a custom
document_idordocument_versionfrom my DynamoDB items, I am not able to keep deletions on my DDB table in sync with OpenSearch, because Data Prepper only reads thenewImageof DDB stream records, and does not write the old imageDescribe the solution you'd like
A new parameter under the
streamblock of DDB sourceWhen this is set to true, Data Prepper would check stream records for the REMOVE action, and then it would construct the Event from the old image, rather than the new empty image.
Describe alternatives you've considered (Optional)
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.