Upgrade compatibility
Understand checkpoint, runtime, and schema support windows
Overview
An API readiness response alone cannot establish that an upgrade is safe. Continuing a session requires compatible database schemas, worker protocols, engine/SDK/CLI/profile digest schemes, and checkpoint formats. Follow the shared pre-upgrade gates for execution order and conditions that prohibit an upgrade.
Support window for current and previous images
The scheduler looks for an image matching the runtime combination recorded in the checkpoint, checking the current image before the previous one. Rebuilds with different digests can still share a runtime combination. If neither image matches, the session enters recovery_required. Workers still running the previous image can create checkpoints with the old combination even after a compatibility check has finished.
Why rolling update order matters
A strict API that does not recognize new fields rejects worker reports. Run migrations and update the control plane before updating workers. Features such as billing:none, session prompts, transcript reasons, and outputs leave data behind once enabled. Disabling a setting therefore does not have the same effect as restoring old binaries.
Checkpoints and restoration
Incremental bundle chains, profile tool lists, and release tool lists are also part of the restoration contract. An older build cannot read an unfamiliar format even if its readiness check succeeds. start_fresh is a recovery decision that discards the previous context; it does not preserve compatibility.
Schemas and running workers
Helm runs migrate before the rollout. Old APIs and workers coexist on the new schema for a time, so changes must expand the schema, switch consumers, then clean up. Changing the worker image does not immediately replace running Pods. Workers whose isolation stamp has changed drain at a turn boundary; turns that exceed the deadline can remain outcome_unknown.
| Change | Running worker Pods | How sessions continue |
|---|---|---|
Worker image (images.worker) | Keep running the old image until they finish | On the next launch, the scheduler checks the new image (images.worker), then the previous image (images.workerPrevious), for a match with the engine/SDK/CLI/profile digest scheme in the checkpoint pointer. If neither matches, the session stops with recovery_required before launch. The Kubernetes pre-upgrade check lists these sessions before the upgrade |
| Isolation settings (stamp values: egress proxy address, storage capacity, worker ServiceAccount, limits, etc.) | Become stale. Claimed Pods drain at a turn boundary and then stop | If the checkpoint covers the last completed turn, a new Pod resumes from the PostgreSQL pointer and S3 checkpoint. If a completed turn lacks a checkpoint, the session stops with recovery_required (CONTEXT_GAP). A turn exceeding the drain deadline (MAX_TURN_SECONDS + 5 minutes) becomes outcome_unknown; its session stops with recovery_required and waits for an operator decision (draining claimed workers) |
| Node replacement or Kubernetes upgrade | Pods disappear. With SIGTERM, they drain and release; without it, heartbeats stop | A released worker is reclaimed immediately. Otherwise, reclamation waits until the scheduler confirms the Pod is gone or the lease expires. If no turn was running and the checkpoint covers the last completed turn, the next launch resumes from that checkpoint. A lagging checkpoint stops the session with recovery_required (session detail attention CONTEXT_GAP). An unresolved turn becomes outcome_unknown, and the session stops with recovery_required (execution backends and worker isolation) |
Checkpoint bucket (config.checkpoints.bucket) | The bucket is part of the stamp, so workers become stale and are replaced | Sessions do not resume automatically. Objects in the old bucket are absent from the new one. If uncollected checkpoint rows exist but the new bucket has no object versions, the api refuses to start (api-server.md). Kubernetes installations have no procedure for moving checkpoints to another bucket. Restoration also refers to versions in the same bucket (backup-restore.md, Kubernetes installations). Do not change buckets while uncollected checkpoints remain |