Define what must survive before choosing technology
A factory connection can fail because of the carrier, router, firewall, DNS, certificates, or the cloud service itself. Machines and gateways may continue operating throughout the outage. The first design question is therefore not how to prevent every disconnection, but which data must be retained, how late it may arrive, and what can be discarded under storage pressure. Alarms, batch genealogy, quality measurements, and cumulative meter readings usually deserve stronger guarantees than continuous raw vibration waveforms.
For each data class, define the tolerated loss, maximum offline horizon, replay deadline, ordering requirement, and local behavior while cloud services are unavailable. Estimate buffer capacity from event rate, serialized event size, and the offline horizon, then reserve space for indexes, retry state, and storage maintenance. Use peak rather than average traffic assumptions. Startup synchronization, batch completion, and bursts of alarms can all raise the ingest rate. Also define an explicit eviction order for a full disk; uncontrolled filesystem failure is not a retention policy.
Build a durable store-and-forward path at the edge
An in-memory queue alone is not outage tolerance. Power loss, process restarts, and container updates can erase it. A safer flow writes each event to durable local storage, acknowledges the source only after that write succeeds, and lets a separate uploader deliver committed records to the cloud. SQLite, an embedded log, or segmented files can all work. The decision should reflect crash recovery, write amplification on the selected storage medium, operational complexity, and the ability to delete acknowledged data safely.
Every event should carry a stable device identity, source timestamp, edge-receipt timestamp, sequence number, schema version, and unique event identifier. Source time helps reconstruct production order, while receipt time exposes clock drift; one should not replace the other. Likewise, a completed HTTP request is not sufficient proof of delivery. Local data should become eligible for deletion only after the cloud has durably accepted it and returned an acknowledgement the edge can reconcile.
- Prefer at-least-once delivery: It is practical across unreliable links, provided cloud writes are idempotent and duplicate event identifiers are ignored.
- Limit ordering scope: Preserve order per device or logical stream instead of forcing the entire factory through one blocking sequence.
- Segment local storage: Time- or size-based segments simplify acknowledgement, cleanup, recovery, and isolation of damaged files.
- Protect capacity: Introduce warning levels, reserve space for critical events, and reduce nonessential sampling before the disk is exhausted.
Control replay instead of flooding the connection
After a long outage, the backlog may be much larger than normal traffic. Replaying it at full speed can saturate the factory uplink and overload brokers, databases, dashboards, or downstream integrations. Schedule live and historical traffic separately, reserve bandwidth for current events, and drain the backlog through bounded batch sizes, concurrency, and rate limits. When the cloud throttles or reports a temporary failure, use exponential backoff with jitter so that many gateways do not retry in lockstep.
Replay order should match the data semantics. Alarms and state transitions may need source-sequence ordering, while independent telemetry can be compressed and uploaded in batches. Cloud consumers must also distinguish event time from arrival time. Reports may backfill historical intervals, whereas an alerting service needs a policy for notifications that arrive too late to be actionable. If schemas change during an outage, the receiver must parse each event by its declared version rather than assuming every queued record has the newest shape.
Make observability and failure drills part of acceptance
A green connectivity indicator does not prove that the pipeline is healthy. Monitor the last device sample, last durable edge write, last cloud acknowledgement, queue depth, age of the oldest pending event, available disk space, retry reasons, and duplicate outcomes. Alerts should distinguish a silent device from a failed edge process, unreachable network, authentication problem, and cloud rejection. That separation gives operations staff an actionable fault domain instead of a generic offline message.
Before production release, disconnect the uplink, block DNS, return cloud errors, restart the gateway, approach the storage limit, and interrupt the connection again while replay is still running. Verify loss and duplication behavior, ordering, restart recovery, and the impact of replay on live ingestion. Offline operation must also preserve security controls: protect local files with access restrictions and appropriate encryption, make certificate-rotation failures diagnosable, and keep device identity stable across reinstalls.
- Verify completeness: Reconcile source sequences or a controlled set of test events with the final cloud records.
- Verify recovery: Observe reconnect, backoff, resume, deduplication, and cleanup as explicit state transitions.
- Verify operability: Confirm that site staff can locate the fault from metrics without manually editing queues or databases.
A resilient pipeline treats offline operation as a normal system state with defined rules for retention, degradation, replay, and cleanup. Where machine protocols, plant networking, cloud models, and ERP operations have separate owners, an experienced integration team can help establish the end-to-end responsibility boundaries.