Define synchronization semantics before choosing pagination
Pagination is not simply incrementing a page parameter. The first design question is what happens when source data changes during a scan. With offset pagination, inserts and deletes can shift later pages, causing duplicate or missing records. Offset pagination is reasonable for small datasets with stable ordering and inexpensive rescans. Frequently changing orders, tickets, or customer records are usually safer with cursor pagination or keyset pagination based on monotonic fields.
A cursor should not be treated as permanent progress. Providers may expire cursors, restrict how long they can be reused, or change their format between API versions. Persist a meaningful business watermark, such as updated_at plus a unique ID, rather than relying only on an opaque next_cursor value. If timestamp precision is limited or several records can share the same update time, sort by a composite key and use both values in the next-page condition.
Before implementation, document the pagination contract:
- Stable ordering:Confirm that every page has an explicit, unique order and determine whether updates can move records between pages.
- Snapshot behavior:Establish whether the scan reads one consistent snapshot or current data on every request.
- Cursor lifetime:Know whether cursors expire, whether they are reusable, and which watermark can restart an expired scan.
- Deletion semantics:Identify tombstones, deletion events, or a full-comparison process for records that disappear.
- Completion signal:Use the documented empty page, missing next cursor, or has_more field instead of guessing from page size.
Treat rate limits as scheduling signals
Immediately retrying an HTTP 429 usually extends congestion. A client should honor Retry-After or the provider's reset timestamp when available. Otherwise, use exponential backoff with random jitter so multiple workers do not wake at the same instant. Configure the retry count, delay ceiling, and overall operation deadline separately. Without an overall deadline, a batch can wait indefinitely while still appearing to be active.
Throughput control should begin before the first 429. A token bucket or explicit concurrency limit can regulate requests, with separate budgets by route, tenant, or credential when the provider applies distinct quotas. Read and write endpoints may not share the same limits, so one global counter is often too crude. If responses expose remaining quota and reset times, concurrency can adjust gradually. If they do not, start conservatively and tune from observed latency, errors, and queue depth.
Not every failure is retryable. Timeouts, connection resets, 429 responses, and selected 5xx errors are often temporary, but retries must remain bounded. Validation failures, permission errors, and malformed requests require correction or isolation. A timed-out write is particularly dangerous because the server may have committed it before the connection failed. Protect retries with an idempotency key, a stable source identifier, or a follow-up lookup that confirms whether the operation already succeeded.
Make bulk synchronization restartable and reconcilable
Treating a large import as one transaction creates an unnecessarily expensive recovery boundary. Divide work into batches with explicit stages: fetch a page, validate and transform it, write it to the destination, confirm the commit, and then advance the checkpoint. The checkpoint must move only after the destination commit succeeds. Recording progress first creates a permanent gap if the process stops before the write completes.
Use upserts keyed by business identity at the destination, and retain a source version, update timestamp, or content hash. Replaying the same batch should produce the same state. When events can arrive out of order, receipt time is not a safe overwrite rule; compare a source version or authoritative business timestamp. Individual invalid records can enter a quarantine queue containing the original input, classified error, and retry state. This prevents one bad record from blocking the entire run without silently discarding it.
- Initial backfill:Build the baseline from a defined boundary and capture the incremental watermark at the start of the scan.
- Incremental catch-up:After the backfill, replay changes from that earlier watermark to cover updates made during the scan.
- Periodic reconciliation:Compare counts, key sets, update ranges, or hashes by partition to detect silent gaps.
- Deletion handling:Prefer deletion events; when unavailable, schedule controlled full comparisons before removing destination records.
Batch size is a trade-off rather than a constant to maximize. Larger batches reduce request overhead but increase memory use, timeout exposure, and replay cost. Smaller batches recover quickly but consume more requests and database transactions. Select a starting size from provider payload limits and destination transaction behavior, then tune it using measured latency and failure patterns. Keep the batch boundary deterministic so an operator can replay one segment without restarting the complete import.
Use observability to prove correctness, not merely completion
A job marked successful only proves that the program reached its final step. Track records fetched, written, skipped, retried, failed, and quarantined for every batch. Expose the current checkpoint, source-to-destination lag, oldest unprocessed update, request latency, and rate-limit waits. Logs should include a sync job ID, tenant, endpoint, cursor summary, and batch ID, while excluding access tokens and complete sensitive payloads.
Alerts should reflect data risk. In addition to repeated failures, detect a checkpoint that stops advancing, a page that replays continuously, new source data accompanied by zero writes, and a growing quarantine queue. Restarting should load the stored checkpoint automatically rather than forcing an engineer to guess the last good page. Operational controls should support replaying a specific batch, acknowledging a confirmed bad record, and initiating reconciliation.
Finally, test recovery through deliberate failures. Stop the worker after a read, time out a destination write, expire a cursor, simulate a 429, and run the same batch twice. The integration is production-ready only when it resumes safely, remains idempotent, and produces consistent reconciliation results. Integrations spanning LINE, ERP, CRM, cloud, and IoT platforms especially benefit from this discipline; when needed, an integration team familiar with both systems can help define the data contract and recovery procedure.