dlt version

1.17.1

Source name

zendesk

Describe the problem

The zendesk source uses time-based incremental loading. This can lead to duplicates as explained in the zendesk docs. The source should be updated to deduplicate adjacent response pages, or use (the zendesk-recommended) cursor-based exports

Expected behavior

The observed behavior can lead to duplicates. data quality issues aside, this also prevents the 'merge' incremental strategy. dlt should either use an incremental method that doesnt have this problem, or auto-dedupe

Steps to reproduce

this might not be straightforward to reproduce since it depends on the specific timestamps of tickets. hmu on slack and i can screenshare if that's helpful! you might be able to check manually with something like postman to check for tickets that exist at both the end of one page and the beginning of the subsequent page

How you are using the source?

I run this source in production.

Operating system

Linux

Runtime environment

Virtual Machine

Python version

3.11

dlt destination

iceberg + filesystem (minio)

Additional information

No response