Treat a command as a state machine, not a packet
A successful MQTT publish, queue write, or gateway API call proves only that one transport step completed. The device may be offline, occupied, rebooting, or unable to satisfy a safety precondition. MQTT QoS can improve delivery guarantees, but it does not make a physical action execute exactly once, nor does it prove that the real-world target state was reached.
Model each command with explicit states such as created, dispatched, accepted, executing, succeeded, failed, and expired, and permit only defined transitions. Store when and why every transition occurred. The API, operator console, alerts, and audit trail can then share one meaning of success instead of forcing engineers to reconstruct events from unrelated logs.
Make every acknowledgment say what it confirms
A generic ACK is too ambiguous for operational control. An accepted response should mean that the device validated and queued the request, not that a valve moved or a firmware update finished. For actions with meaningful duration or risk, separate receipt acknowledgment from completion acknowledgment. Longer operations may also emit progress or heartbeat events. Responses should carry the command ID, device ID, current state, device timestamp, error code, and a machine-readable reason.
Use layered deadlines rather than one global timeout. A dispatch timeout tests whether the broker or gateway accepted the work. An acknowledgment timeout indicates that the device has not accepted or rejected it. An execution timeout says that completion exceeded the expected operating window. Set these values from the device connection model, normal duty cycle, sleep behavior, and consequence of duplicate execution.
- Safe to retry: State queries and idempotent configuration updates can usually use bounded retries with backoff.
- Query before retrying: Door release, dispensing, and machine-start commands should be reconciled with the original result first.
- Do not retry blindly: Irreversible calibration, metering, or financial side effects need compensation or human review.
- Keep late results: A success received after timeout is evidence, not noise; record it as a late outcome.
Deduplicate where the side effect actually occurs
Duplicate delivery is a normal operating condition. Reconnects, broker redelivery, consumer recovery, and repeated user input can all reproduce the same intent. Generate a stable commandId when the command is created and reuse it for every retry. The device, or the gateway nearest to it, should deduplicate on the combination of device identity and command ID.
API-level deduplication alone cannot prevent a repeated physical action after a message has left the API. Retain deduplication records long enough to cover the longest credible offline, retry, and delivery-delay window. Store the prior outcome with the key so a duplicate can receive the same answer without executing again. A constrained device may use a persistent ring buffer or delegate the record to a gateway, but critical keys must survive restarts.
- Same ID and same payload: Return the stored state or result without executing again.
- Same ID and different payload: Reject it as a protocol error; never let an ID collision hide a new action.
- Different IDs with one business intent: Add a domain idempotency key, such as a schedule run and target state.
- Prefer desired state: “Set output to off” is generally safer to retry than “toggle output.”
Design for uncertainty and make it observable
A distributed control path cannot eliminate every unknown outcome. When the cloud times out, the device may not have received the command, may still be executing it, or may have completed it while the reply was lost. Therefore, expired or unknown must not automatically mean failed. Provide result queries, replay final outcomes when a device reconnects, and define domain-specific compensation. If a newer command supersedes an older one, specify version ordering, priority, and when cancellation becomes too late.
Before release, inject disconnects, delays, reordering, duplicate delivery, device restarts, and cloud-consumer restarts. Observe time spent in each state, retry counts, late outcomes, deduplication hits, invalid transitions, and device error codes. These signals let operators decide whether to wait, retry, reconcile current state, or intervene.
- Classify every command by side effect, retry safety, and completion condition.
- Carry one correlation identity across the API, broker, gateway, and device.
- Require devices to report final outcomes and answer queries for prior commands.
- Version timeout policies and deduplication retention rules with the command protocol.