Define what “out of sync” actually means
When a user reports that an order is shipped in the ERP but still processing in the CRM, resist the urge to replay the integration immediately. First establish whether both systems are describing the same business concept. A commerce platform may expose only processing, shipped, and completed, while a warehouse distinguishes picking, packing, carrier handoff, and delivery. Finance may care about invoicing, settlement, and refunds instead. Two labels can differ without either record being wrong.
Capture the order identifiers, the current value in every system, the expected value, and the business event that should have caused the transition. Then determine whether the disputed state belongs to the whole order, an individual line, a shipment, a payment, or a return. Split shipments and partial cancellations are especially important: an integration that stores only an order-level status may collapse valid detail into an incorrect summary.
- Identity: Verify the relationships among customer order numbers, internal IDs, shipment IDs, invoices, and return references.
- Semantics: Compare entry conditions and terminal conditions, not merely similar-looking status labels.
- Time: Separate the business-event time from source publication, middleware receipt, and target write times.
- Scope: Identify whether the value describes an order, line item, payment, invoice, fulfillment, or delivery.
Preserve evidence and follow the event trail
Avoid editing the database or launching a bulk replay before collecting evidence. Those actions can replace useful timestamps, generate duplicate notifications, or erase the path that produced the mismatch. Build a timeline instead: when the source committed the change, which component read it, what payload it emitted, how middleware transformed it, what the target returned, and whether a retry, scheduled job, or manual action followed.
Every hop should carry a correlation identifier that can be searched across services. If the current architecture has no trace ID, correlate temporarily using the order ID, event type, source timestamp, and payload hash. Treat that method cautiously: identifiers may be reused, and timestamps may differ because of time zones or clock drift. Logs should retain direction, endpoint, response status, business error code, attempt number, and processing result. A message such as “request succeeded” does not prove that the target accepted the business transition.
Reconstruct transitions, not just final values
Once the records are collected, order them by business causality rather than database write time alone. Typical breakpoints include late webhooks, duplicate queue delivery, batch imports overwriting newer values, missing status mappings, and races between manual work and automation. A cancellation may reach the target first while an earlier shipped event is delayed and retried later. Without a version check, that older event can move the target into a state that no longer reflects the business reality.
For each hop, ask a precise question: did the source publish the event, did transport deliver it, did transformation preserve its meaning, did the target accept it, and did another process overwrite the result? Keep technical success separate from business success. An HTTP 200 response may mean only that a request was queued for background processing. Conversely, a timeout does not prove that the write failed; an automatic retry may create a second shipment if the endpoint is not idempotent.
- Missing events: Check for a gap between committing the transaction and publishing its event; an outbox pattern can close that gap.
- Duplicate events: Use a stable event key and durable idempotency record, not only a short deduplication window.
- Out-of-order delivery: Compare business versions or sequence numbers instead of trusting network arrival order.
- Transformation defects: Retain both the source payload and mapped output so the conversion can be reproduced.
- Later overwrites: Inspect batch synchronization, imports, scheduled jobs, and other integrations that write the same fields.
Assign authority before repairing data
The central responsibility question is who is allowed to declare a state valid. The answer is rarely that one platform owns every order field. Payment authority may belong to the payment or finance system, fulfillment authority to the warehouse, and customer contact details to the CRM. If multiple applications can update the same field, define precedence, permitted transitions, and what happens during conflict: reject the update, hold it for review, or accept it only when its version is newer.
Choose remediation according to scope and reversibility. A confirmed isolated mismatch can be corrected with a controlled compensation command. For recurring or widespread discrepancies, pause the faulty write path first and compare downstream records with the authoritative source. Replacing an entire target table may be fast, but it can destroy valid local adjustments. A safer reconciliation process produces a reviewable difference set and applies changes by field, version, and allowed transition while preserving an audit record.
Replay also requires judgment. Reusing the original event preserves its meaning but may retrigger emails, inventory reservations, or partner callbacks. A dedicated repair command can suppress inappropriate side effects, although it introduces another code path that must be tested and governed. For high-impact operations, prefer a dry run that shows the intended changes, then execute with stable idempotency keys and explicit approval boundaries.
Turn the incident into an operating control
After recovery, add three controls: a status contract, end-to-end traceability, and scheduled reconciliation. The contract should name the authority for each field, mapping rules, irreversible states, version behavior, and exception paths. Monitoring should cover queue age, processing lag, dead letters, rejected transitions, and repeated retries rather than checking only whether services are running. Reconciliation should compare authoritative records with downstream snapshots and produce a replayable difference set instead of asking operations staff to inspect screens manually.
Useful alerts should point toward ownership. “Orders differ” is less actionable than identifying the source system, affected transition, oldest pending event, last successful hop, and component responsible for the next action. Retain enough payload history to investigate safely, but redact credentials and unnecessary personal data. Traceability is valuable only when its access and retention rules are deliberate.
A mature integration does not promise perfect agreement at every instant. It defines acceptable delay, conflict behavior, recovery procedures, and the evidence needed to explain each record’s path. Once the responsibility chain is searchable, replayable, and auditable, order mismatches become bounded engineering problems instead of recurring disputes between business teams.