Introduction
If you see duplicate rows in an append or incremental extract, this is often expected behaviour for certain connectors. In most cases, the source system or API returns overlapping records between runs, and Kleene loads those records into the raw table as received.
Why duplicate rows happen
Append and incremental extracts are designed to capture new and changed data since the last run. Some source APIs do not return a perfectly non-overlapping set of records. Instead, they may include records that were already returned in a previous extract, especially when timestamps, pagination, or sync windows overlap.
Kleene loads the extracted data into raw tables as-is. That means duplicate rows can appear in the raw layer even when the connector is working correctly. This is normal for connectors that rely on source-side incremental logic rather than strict row-level deduplication at extract time.
What to do
The recommended approach is to de-duplicate in the transform layer, not at the extract level. Build a transform that removes duplicate rows and keeps the most recent record for each business key or unique identifier.
A typical transform pattern is to sort records by the relevant timestamp or version field, then keep only the latest row per key. This ensures your downstream models use the most up-to-date record while preserving the raw source data for traceability.
Example de-duplication approach
1. Identify the unique key for the record, such as customer_id, order_id, or another source identifier.
2. Choose the field that defines the most recent version of the record, such as updated_at, modified_at, or synced_at.
3. Group records by the unique key and retain the row with the latest timestamp or highest version.
4. Use the de-duplicated output in your downstream reporting or analytics models.
When to check the connector setup article
If you are unsure whether duplicate rows are expected for your connector, check the connector's setup article first. The article may explain known incremental behaviour, overlap windows, or fields that should be used for de-duplication.
When to contact Kleene Support
Raise a support ticket with Kleene Support if the duplicate rows do not match the connector's documented behaviour, or if you believe the extract is returning unexpected data. Include details such as the connector name, extract type, example duplicate rows, and the fields used for incremental loading.
Summary
Duplicate rows in append and incremental extracts are often caused by overlapping records returned from source APIs. Kleene loads raw data as received, so the correct place to remove duplicates is usually the transform layer. If you are unsure whether the behaviour is expected, review the connector setup article or contact Kleene Support for help.