Separate event time from processing time
Event time records when a transaction, device reading, or user action actually occurred. Processing time records when the platform received or computed it. Offline applications, buffered IoT gateways, scheduled API exports, network retries, and queue backlogs can create a substantial gap between the two. If aggregates use processing time, an event from yesterday that arrives today is credited to today, distorting daily reports, billing periods, and anomaly detection.
We normally require every event to carry a stable event_id, event_time, ingest_time, source identity, and version information. The system closest to the business fact should produce event_time, which should be stored in UTC; reporting can then apply an explicit business timezone. If a source clock is unreliable, do not silently substitute receipt time. Record the clock offset, inferred-time rule, or a quality flag so downstream consumers understand what the timestamp means.
Use watermarks to close periods deliberately
A watermark represents the point in event time before which the system believes most expected data has arrived. It is not simply the greatest timestamp observed, and a fixed delay is not automatically a sound policy. Set allowed lateness from observed source behavior, retry patterns, network conditions, and the business deadline for results. Longer waits improve completeness but delay output and retain more state; shorter waits provide faster numbers but create more corrections.
The closing policy can differ by use case. An operational dashboard may publish provisional figures, while billing or finance may require controlled close and adjustment workflows. Before implementation, decide:
- Accepted lateness:How late is normal for each source, and when should automatic acceptance stop?
- Window boundaries:Are totals grouped by hour, calendar day, shift, or accounting period, and in which timezone?
- After-close behavior:Will a late event reopen the window, update an aggregate, create an adjustment, or enter manual review?
- Result status:Does a consumer receive a provisional value, the latest revision, or a formally closed version?
Make correction and deduplication end-to-end properties
A watermark controls computation timing; it does not eliminate duplicates. Producer retries, message redelivery, and job recovery can present the same event again. Deduplication therefore needs a stable key, and sink writes must be idempotent. Mutable business events also need a source version, sequence number, or validity interval. Arrival order is not version order: an older update may arrive last and must not overwrite a newer state.
Keep an immutable raw event record, then derive queryable current state in a curated layer. If late data can change an aggregate, the destination must support upserts, retractions and recomputation, or append-only adjustment entries. An append-only total without correction semantics usually double-counts. For a formally closed period, preserve the original settlement version and add a traceable adjustment instead of rewriting history. For an open operational period, recomputing the affected partition is often simpler. Batch backfills and streaming jobs should share transformation rules so recovery does not produce a second interpretation of the data.
Verify the policy through reconciliation and observability
A late-data design must be observable in production. Measure the difference between event_time and ingest_time by source, along with beyond-watermark events, duplicates, reopened windows, adjustments, and timestamps that cannot be parsed. Pipeline success alone is weak evidence: every job can be green while records are assigned to the wrong business day.
Tests should cover window boundaries, day and month transitions, clock skew, duplicate delivery, late older versions, recovery after an outage, and replaying a job. In production, reconcile source detail, immutable events, and aggregates on a regular schedule. Each published result should be traceable to its input events, transformation version, and settlement version. A mature event-time design does not assume data will always be punctual; it ensures that when delays occur, corrections are consistent, explainable, and reproducible.