Introduction
Append Overwrite is a load method designed to help when recent data is missing, duplicated or needs to be refreshed. In Kleene, it works by overwriting data from a recent timeframe based on a configured offset, then loading the latest data again.
This guide explains what append overwrite does, when it is useful and when another approach may be better.
Issue description
Recent data can sometimes be updated, corrected, duplicated or missing in the destination table. A standard append load may add new rows without correcting recent records that have changed. Append Overwrite helps by replacing a recent window of data so the latest source values can be reloaded.
However, append overwrite is not a smart record-level merge. It does not selectively update only changed records for each ID. It overwrites a timeframe and reloads that period.
Signs
Append overwrite may be useful if:
- Recent records are missing from the destination.
- Recent records are duplicated.
- Source data changes after it is first extracted.
- A recent reporting window needs to be refreshed each run.
- Append-only loading is leaving stale values.
- The source updates records within a predictable recent timeframe.
It may not be the best option if updates happen across a wide or unpredictable historical range.
Depending on how often records are updated, append incremental followed by partitioning or deduplication in a transform may be a better option.