Separate a broken connection from a failed job
When a browser stops receiving tokens, the backend has not necessarily stopped working. A mobile network change, proxy timeout, page refresh, or closed tab may terminate the transport while the model request, retrieval pipeline, or tool call continues. Treating every disconnect as a failed job creates expensive ambiguity. An automatic retry may start a second model invocation, write the same CRM note twice, or leave the original generation consuming resources with no subscriber.
Model each response as a durable job with a stable request ID, and track transport state separately from execution state. The job should have explicit states such as queued, running, cancelling, completed, failed, and cancelled. A disconnected subscriber does not get to declare the job failed. Each emitted event should also carry a monotonically increasing sequence number. This gives the client a precise recovery position instead of forcing it to infer progress from the amount of text currently displayed.
Propagate cancellation across the full call chain
A Stop button is useful only when cancellation travels beyond the interface. The client should issue an authenticated cancellation command for the request ID. The API then signals the worker, which passes cancellation through retrieval, model streaming, database operations, and internal tools wherever those components support it. If a third-party service cannot cancel an in-flight call, the job can enter cancelling and discard later output, but the system should not claim that upstream computation stopped immediately.
Completion and cancellation can race, so state transitions must be atomic. Define which outcome wins before implementing the endpoint. A practical policy is that a persisted completed result remains completed, while a running job that has not committed its result may accept cancellation. Tool side effects require another distinction: stopping future steps is not the same as reversing completed work. Discarding an email draft is simple; retracting a sent notification or deleting an ERP record may require a separately authorized compensation workflow.
Resume from an event log, not from a partial string
Reliable resumption requires a server-side history of acknowledged output. When reconnecting, the client sends the request ID and its last acknowledged sequence. The server replays missing events and then switches the subscriber to live delivery. If the job has already completed, it returns the persisted final response. If the event retention window has expired, it should return an explicit resync_required result and provide a current snapshot. This protocol prevents duplicate text and avoids rerunning a model simply because a network connection changed.
- Brief transport loss:keep the job running and replay events after the last acknowledged sequence.
- Worker or server restart:recover job state, committed events, and tool records from shared storage rather than process memory.
- Model-provider interruption:if the provider cannot resume the same generation, close the original attempt as failed and create a new attempt.
- Completed tool call:look up the prior result by idempotency key instead of executing the side effect again.
- Expired event history:send a full response snapshot and instruct the client to replace its local projection.
Avoid feeding an unfinished paragraph back to a model with an instruction to continue from the last character. The model may repeat text, change its reasoning, or lose the tool context that produced earlier claims. If the product genuinely needs continuation across processes or providers, create a structured checkpoint containing confirmed output, citations, tool results, conversation state, and remaining objectives. Expose the continuation as a new generation attempt, even if the interface presents one logical answer.
Make retry and recovery explicit in the state machine
The frontend should project authoritative server state instead of treating a loading boolean as truth. After a reload, it first queries the request, then decides whether to reconnect, display a completed answer, offer a retry, or warn that an external action may already have occurred. A retry should receive a new attempt ID while retaining the same business-level idempotency key. That preserves diagnostic history for each model call without allowing a reconnect or retry to repeat a write operation.
Long-running jobs also need leases, heartbeats, and clear worker ownership. A replacement worker should not assume ownership merely because a heartbeat arrived late; it must acquire a lease atomically and inspect any committed steps. Store enough context to distinguish a worker that died before a tool call, during the call, or after the result was recorded. For external systems that offer their own idempotency tokens, pass the business key through rather than relying solely on a local database lock.
Test recovery by interrupting real boundaries, not only by mocking a successful stream. Close the browser between tokens, cut the connection after a tool returns, restart a worker, delay a cancellation command, and make reconnection race with completion. Acceptance criteria should verify that content is not duplicated, only one terminal state is committed, cancellation remains auditable, tool side effects are committed at most once, and expired event logs converge on a consistent snapshot. Once these behaviors are encoded as state-transition and contract tests, streaming becomes an operable enterprise capability rather than a fragile presentation effect.