Define the intended overlap behavior first
Job overlap is usually a symptom, not the root problem. A run may exceed its interval because the data set grew, an external API slowed down, a database statement waited on a lock, or a retry added unexpected time. Overlap also appears when several application instances receive the same schedule event. Before choosing a locking library, decide what the business expects when a second trigger arrives.
The valid choices are typically to skip the new run, wait for the current run, queue every run, or replace an older run with one based on fresher data. A periodic snapshot may safely skip intermediate executions. A reconciliation process may need every period to be completed eventually. Payments, notifications, order creation, and integration events usually require each logical item to be processed once, even if the surrounding job is triggered more than once.
Mutual exclusion prevents simultaneous execution; idempotency prevents duplicated outcomes. They are not interchangeable. A worker can finish its external side effect and crash before recording success, causing the scheduler or queue to deliver the work again. Important operations therefore need an idempotency boundary even when a distributed lock is already present.
- Skip: use when only the freshest state matters and missing one interval is acceptable.
- Wait: use when every run matters but delayed completion is acceptable.
- Coalesce: use when several triggers can be represented by one refreshed calculation.
- Process each item: use for orders, financial actions, notifications, and durable integration events.
Coordinate where every worker can see the decision
An in-memory flag or process mutex can work in a genuinely single-process service. It stops protecting the job as soon as the service scales horizontally, restarts during deployment, or receives triggers from more than one scheduler. Coordination state should live in a system shared by all possible workers, such as the primary database, Redis, or a durable queue with controlled consumer concurrency.
Choose the mechanism by examining the infrastructure already operated by the team and its failure modes. A conditional database update, unique constraint, or advisory lock is often a good fit for low-frequency jobs and can sometimes share a transaction boundary with business data. Redis provides fast lease-based coordination, but ownership, expiry, and network partitions require care. A queue can absorb bursts and serialize consumption, but it may redeliver messages and does not make the business operation idempotent by itself.
Lock granularity determines both safety and throughput. One global key for an entire job is easy to reason about, but it can unnecessarily block unrelated tenants, dates, or partitions. A more scalable key combines the job type with the relevant business scope, such as tenant and settlement date. Finer keys improve parallelism while increasing the complexity of naming, monitoring, and recovery.
- Build lock keys from stable business identifiers, not process IDs or random values.
- Acquire the lock atomically; a separate read followed by a write still has a race condition.
- Store an owner token and allow only that owner to release or renew the lock.
- Treat duplicate-key conflicts used for deduplication as an expected control path, not an unknown system error.
Design leases and retries for failure, not the happy path
A lock without an expiry can remain forever after a worker crashes, so distributed locks commonly use leases. A lease that is too short may expire while useful work is still running, allowing a second worker to enter. A lease that is too long delays recovery after a genuine failure. Base the duration on realistic worst-case behavior, including data-volume spikes, API latency, runtime pauses, and the shutdown window during deployment—not merely the average run time.
Long-running jobs can renew their leases with a heartbeat, provided every renewal validates the owner token. For operations where stale writers would be dangerous, add a monotonically increasing fencing token. Each new lock holder receives a newer token, and the downstream write accepts only the newest one. If an old worker resumes after a network pause and its lease has expired, its stale token prevents it from overwriting work completed by the new owner.
The final safeguard belongs at the business boundary. Assign a stable execution key to each logical unit of work and check it when creating side effects, using a unique constraint, a conditional state transition, or an outbox record. When an external provider supports idempotency keys, send that same stable key on retries. When it does not, persist the request and known result locally, then provide a reconciliation path for cases where the remote action succeeded but the local acknowledgement was lost.
- Record the execution key, lock key, trigger source, and attempt number before processing.
- Checkpoint large batches so a retry can resume without replaying completed partitions.
- Use explicit state transitions, such as allowing only pending items to become processing.
- Do not blindly resend an irreversible external operation when its previous result is uncertain.
Make skipped, retried, and stuck work visible
A deduplication mechanism without observability can hide a real outage behind apparently normal skips. Each trigger should expose its scheduled time, actual start time, lock wait, execution key, owner, processing scope, completion state, and failure reason. Metrics should distinguish successful runs, overlap skips, lock timeouts, successful retries, and terminal failures. Otherwise, operators may see the scheduler firing normally while the underlying data remains stale.
Alert on business freshness rather than every isolated overlap. A frequent synchronization job may legitimately skip a trigger because the previous run is still active. The actionable condition is often that no run has completed within the business’s acceptable freshness window. Manual replay tools must use the same lock keys and idempotency rules as scheduled execution; an administrative button should not become an unprotected second path into the workflow.
Before release, deliberately test two workers starting together, the lock holder terminating mid-run, a lease expiring at the boundary, an external call succeeding before local state is saved, and a queue message being delivered again. A reliable design does not promise that triggers will never be duplicated. It makes duplicate delivery predictable and recoverable while ensuring that the same business outcome is not created twice.