Timeouts, Retries, and Idempotency: Preventing Duplicate Work

A timeout tells the caller that it did not receive a result within its waiting budget. It does not prove the server did nothing. Before retrying a request with side effects, establish an operation identity that the server can recognize across attempts and define what happens when the first attempt has already committed.

Use bounded retries for documented transient failures, share one overall deadline across attempts, and keep the same idempotency key for the same intended operation. If the service cannot resolve the uncertainty safely, expose an unknown outcome and a reconciliation path instead of silently creating another operation.

The response can disappear after the work succeeds

Imagine an export API. The client requests a new export job. The server inserts job 41 and commits, but the response is lost. The client sees a timeout and sends another creation request. Without a shared operation identity, job 42 looks like a legitimate new request.

The problem is not that the retry is too fast. A delay of ten seconds would still create a duplicate if the second request represents a new operation to the server. Backoff controls pressure; idempotency controls repeated effects.

RFC 9110, section 9.2.2 defines idempotency in terms of the intended server effect. PUT, DELETE, and safe methods have idempotent semantics; a client should not automatically retry a non-idempotent request without knowing its semantics permit it or knowing the original was not applied. A POST endpoint needs an explicit application contract before an automatic retry can rely on this property.

Give one intent one identity

For the export API, the client creates an opaque key once, before its first attempt. The server scopes that key to the authenticated account and operation type. Alongside it, the server records the relevant input or its canonical fingerprint and the durable result.

The desired decisions are:

Hashing the request body alone is usually insufficient to identify intent. A user may legitimately request the same export twice. Conversely, insignificant serialization differences should not necessarily create a new operation. The explicit key distinguishes an intentional second operation from a repeated delivery of the first.

The Amazon Builders' Library treatment of idempotent APIs explains caller-provided request identifiers and the need to keep recording that identity consistent with the mutation. If a service creates the job and crashes before recording the key, the duplicate window remains.

What the local fixture actually proved

We executed a Python and SQLite example that inserts a synthetic export job, commits, and then deliberately raises TimeoutError before returning the result. The client retries. The keyed path records the request and job in one transaction; each attempt opens a new connection to a temporary database file.

The observed results were:

Without a key: two jobs after two attempts.
With the same account and key: one job after two attempts.
Same key with different input: key_payload_conflict.

The core transaction is small enough to describe directly:

begin transaction
look up (account, operation key)
if found:
    reject a changed input
    return the recorded job identifier
otherwise:
    insert the job
    insert the request identity and job identifier
commit before returning the response

SQLite's write transaction serialized the fixture's mutation. The example did not test distributed workers, concurrent HTTP requests, a process crash, or a downstream service. Its timeout is an injected exception after a real local commit, not an observed network failure. It demonstrates the ambiguous-result case without pretending to qualify a production implementation.

For concurrent requests, make uniqueness and transaction behavior part of the storage design. Two workers must not both pass a non-atomic “key absent” check and create separate jobs. For an external effect, such as sending email, a local transaction alone cannot atomically commit the remote system's work. Use the downstream provider's supported idempotency contract or a delivery/reconciliation design that explicitly handles that gap.

Budget attempts against the caller's deadline

Suppose an interactive caller allows two seconds in total. Giving each of three attempts a two-second timeout violates that caller's budget even before backoff. Track an absolute deadline with a monotonic clock. Before each attempt, calculate the remaining time and cap both the next call and its delay accordingly.

Distinguish connect timeouts, response-read timeouts, and overall deadlines in the client you actually use. Do not assume a setting called timeout covers DNS, TLS, queueing, streaming, and all retries. The Amazon guidance on timeouts and jitter explains why retries can amplify overload and why spreading attempts helps avoid synchronized bursts.

Choose a retry owner. If the application, SDK, proxy, and worker each make three attempts, one user action can produce far more than three downstream calls. Record the SDK's attempt semantics and decide which layer owns the budget. The article's local example intentionally has two attempts and no delay; it tests deduplication, not a tuned backoff policy.

Classify failures from the API's contract. An invalid input generally needs correction. An authorization failure needs the documented credential-recovery path. A rate limit may carry a wait instruction that still has to fit the deadline. An unknown server outcome needs the stable key or a lookup operation. A blanket retry of every exception erases these distinctions.

When the budget ends, preserve the operation identifier and communicate the uncertainty. “We could not confirm completion; check job 41” is recoverable. “Failed, try again” can encourage the user to initiate a second operation while the first is still running.

The invariant to test is concrete: repeated delivery of one authorized intent must not create additional intended work within the service's documented idempotency window. Timeouts and retries are useful only when that contract remains true under the failures your system actually faces.