Graceful shutdown should stop admitting new work, give accepted work a bounded chance to finish, and leave unfinished work in a state another process can understand. A termination signal starts that protocol. It does not prove that requests completed, responses arrived, or queue messages can safely be acknowledged.
Design the deadline and recovery behavior together. If a job can outlive the process's shutdown budget, the system needs cancellation, a durable checkpoint, or a safe retry mechanism. An indefinitely longer grace period only postpones the unresolved decision.
Follow one request through the shutdown boundary
Imagine an export service that has accepted a job and is writing its result when a deployment replaces the instance. At least four events matter: the instance stops receiving new traffic, the process receives a stop request, accepted work finishes or is interrupted, and the supervisor eventually enforces its deadline.
These events are related but are not one atomic operation. Kubernetes describes termination as concurrent local shutdown and EndpointSlice updates. A configured preStop hook consumes the termination grace period; the runtime then sends the stop signal, and remaining processes can be killed when the budget expires. Terminating endpoints report ready: false, while connection draining can use additional endpoint state. See Pod termination flow.
For application design, assume some traffic can still arrive while routing information propagates or an established connection remains open. The application must enforce admission locally. A routing update is not proof that its last request has already arrived.
Give the process an explicit draining state
Use a small lifecycle with clear transitions:
serving
-> draining: close admission, stop fetching new jobs
-> finished: accepted work is complete; release resources; exit
-> deadline reached: interrupt or leave recoverable work; exit
The transition to draining should be idempotent. A second stop request must not reset the deadline and extend shutdown forever. Capture the active-work set under the same synchronization used to admit work; otherwise a request can slip between “stop accepting” and “count what remains.”
For an HTTP server, closing the listening socket and handling established connections depend on the framework and protocol. An open keep-alive connection is not the same as one active request, and streaming protocols can remain active for a long time. Check the framework's drain behavior rather than assuming a server-close method covers every connection type.
For a queue worker, stop requesting new messages before waiting for active handlers. A message already delivered to the process remains its responsibility until it is durably completed, safely released, or allowed to become available according to the broker's lease rules. Acknowledging early to make shutdown look clean can turn an interruption into lost work.
Keep signal handling small
The signal handler should request shutdown. Put the draining procedure in the normal event loop or lifecycle mechanism, where it can coordinate with workers and resource owners. Avoid performing unbounded I/O inside the handler.
Our local Python fixture uses this pattern:
stopping = False
def on_term(signum, frame):
global stopping
stopping = True
signal.signal(signal.SIGTERM, on_term)
The complete fixture uses the same flag inside a child function. Its main loop observes the flag, sets a monotonic deadline once, and reports that admission is closed. Python's signal documentation explains that Python handlers run in the main thread and are executed by the interpreter rather than directly inside the low-level handler. Long-running native work can therefore affect responsiveness. This is not a portable shutdown implementation for every runtime.
Closing admission is a modeled event in this fixture: there is no network listener or queue broker. The signal delivery and child-process termination are real.
Three local outcomes, including forced interruption
We ran three POSIX subprocess cases with Python 3.14.2 on macOS. The parent waited for job_started, sent SIGTERM, and waited for the child's admission_closed event. The child used a 0.25-second application drain budget. These deliberately small durations are fixture inputs, not recommended production settings or timing measurements.
| Case | Observed events after the job started | Process result |
|---|---|---|
| Short work, configured for 0.1 seconds | admission_closed, then job_completed |
Exit code 0 |
| Long work that honors the drain deadline | admission_closed, then job_interrupted_at_deadline |
Exit code 2 |
| Long work that ignores the drain deadline | admission_closed; no completion event |
Parent killed it after its wait expired; return code -9 |
The parent used a 0.6-second wait after observing admission closure before forcing the final case to stop. It verified the expected return codes and whether a completion event existed. All three cases passed their assertions.
This proves that the synthetic worker can complete, report interruption, or be forcibly killed under the programmed conditions. It does not prove that an interrupted operation was rolled back, that a response reached a client, or that a real queue would redeliver the job. The work is a timed loop with no external effect. The absence of job_completed is a fixture observation, not a general method for deciding whether an arbitrary remote operation committed.
Completion and acknowledgment are different facts
Suppose an export's database record commits, but shutdown cuts the connection before the client receives the success response. The client may retry a job that already exists. Graceful shutdown reduces the chance of this outcome; it does not remove the uncertainty window.
Give the operation a durable identity and a reconciliation path. Our timeouts and idempotency guide demonstrates this distinction with a local commit followed by a simulated lost result. The same reasoning applies to a worker that completes an external effect and stops before acknowledging its message.
Checkpointing also needs an honest contract. “Last processed row 500” is useful only if the checkpoint and corresponding effects agree. A checkpoint written before the work can cause skipped work after restart; one written afterward can permit repetition. Choose transactions, idempotent effects, or reconciliation based on the boundary involved.
Budget shutdown from the outside inward
Identify the supervisor's deadline, any hook time, application drain time, resource cleanup, and a margin for scheduling uncertainty. The application should normally finish its own bounded procedure before the supervisor must force termination. An illustrative 30-second external budget with a 20-second application drain is an arithmetic allocation, not evidence that a workload fits it.
Measure shutdown with representative work: a short request, a slow dependency, an idle connection, an active stream if supported, and a job interrupted around its durable commit. Record which jobs completed, which were recoverable, and which outcomes remained uncertain. Confirm that the real main process receives the signal and that wrappers forward it correctly.
For diagnosis, preserve the reason for shutdown, active-work count, deadline, and final outcome without recording sensitive payloads. Connect these events to request and job identities using the approach in our observability guide.
A successful shutdown is a bounded, explainable outcome for accepted work. A process disappearing from the deployment is only the last visible step.
Documentation and the local subprocess exercise were checked October 28, 2026. No Kubernetes cluster, production service, HTTP drain, or queue acknowledgment was tested.
