Model the workflow as a state machine, not a chain of API calls
A typical automation may create a form, write a record to an ERP, start an approval, update a CRM, and send LINE or email notifications. If the implementation simply calls each API in sequence, one timeout creates an ambiguous outcome: the request may have failed before reaching the service, or the service may have completed the operation and lost the response. Restarting the entire sequence can then create duplicate documents, parallel approvals, or repeated notifications.
Assign every workflow instance a stable business identifier and persist its current state, attempt count, external identifiers, and last error. States should represent verifiable facts such as ERP document created or approval result unknown, rather than a vague processing flag. A restarted worker can then resume from the last confirmed boundary instead of reconstructing intent from logs.
Separate retry, compensation, and human intervention
Not every error deserves a retry, and not every completed action can be rolled back. Classify each step by side effects, reversibility, query support, and business risk before choosing its recovery policy.
- Transient failure: Retry network errors, rate limits, and temporary outages with backoff and a firm attempt limit. The operation must be idempotent first.
- Deterministic rejection: Validation errors, missing permissions, and invalid approval rules require correction. Repeating the same request only creates noise.
- Unknown result: After a timeout, query by business key or request identifier before issuing another create operation.
- Reversible side effect: Cancel, void, or delete an unused draft when the target API supports it, while retaining an audit record.
- Irreversible side effect: For an approved decision, read notification, or posted accounting entry, create a correction, revocation, or reversing transaction instead of pretending the event disappeared.
Combine idempotency, an outbox, and Saga compensation
Idempotency keys protect form creation, approval submission, and notification delivery. Repeating the same business operation should return the original outcome. If a target API lacks native idempotency support, the integration layer can store a mapping between its business key and the external record identifier, then query before and after creation. Include the operation and version in the key so that a legitimate resubmission of a revised request is not suppressed as a duplicate.
A transactional outbox addresses the dual-write gap between updating a local database and publishing an event. Commit the business change and pending event in one local transaction, then let a worker deliver the event. Across systems, use a Saga with an explicit compensation for each completed step. If an ERP document exists but approval creation fails, the compensation might mark the document as blocked or void rather than deleting it. Because compensation can also fail, it needs its own idempotency key, retry policy, and terminal state.
Approvals and notifications need different semantics
Approvals contain human decisions and an audit trail. If downstream synchronization fails after submission, the safer response is usually to pause further processing, flag the inconsistency, and preserve the history. When form data changes during approval, create a new version and request approval again; silently replacing the content breaks the relationship between the decision and what the approver actually reviewed. Cancellation should record its reason, actor, and superseded version.
Notifications are usually impossible to retract. Their compensation strategy is to prevent duplicates, avoid announcing success too early, and send an explicit correction when necessary. Identify a delivery by recipient, template version, business event, and channel. If a LINE message succeeds while email fails, retry only the email delivery. A global replay that broadcasts every channel again is operationally simple but produces confusing user behavior.
Make failure operable, not merely observable
An operations view should expose the business identifier, current step, external identifiers, error class, next retry time, and compensation status. Alert on conditions that require intervention, such as exhausted retries, contradictory external data, or failed compensation, rather than every temporary timeout. A permission-controlled console should support actions such as retrying one step, confirming an external outcome, running compensation, and closing a case with a reason.
Before release, inject failures at every boundary: drop the response after the target succeeds, restart a worker before event publication, make the compensation API fail, and replay the same event. The design is ready when it can recover from persisted state and an operator can explain why each item stopped where it did. For workflows spanning several legacy or third-party systems, agreeing on states, ownership, and recovery boundaries with an experienced integration team is often the most valuable part of the implementation.