Upgrade a single server

What you will doCopy Anchor Link

Upgrade Compose after backup and compatibility checks. Approve checkpoint loss only after identifying the exact affected sessions.

Before you beginCopy Anchor Link

Read the target version's operator actions in the changelog.

You need current/target source commits, release inputs, settings digests, and image digests. Check a recent successful backup and restoration drill, and notify users of downtime. This full Compose upgrade has not been verified with published images.

Check the shared pre-upgrade gates first. On LocalStack trials, the first restart-policy change can recreate the container and lose checkpoint objects. Postpone unless status confirms 0 uncollected checkpoints. API startup failure with pending.json cannot be bypassed through reseed. The remaining escape, reset, discards all data: assess backup restoration first. backup --stop does not remove this block.

StepsCopy Anchor Link

1. Target input and backupCopy Anchor Link

Select the tag from the package contract. Generate release-new.json outside the checkout as in installation release generation. First check out the target source commit and run bun install --frozen-lockfile. Release source_commit must equal checkout HEAD. Perform a backup first.

2. API → proxy → worker orderCopy Anchor Link

For protocol or gate field changes, do not replace every image together. Check out the new source commit and install dependencies. Create an intermediate release changing only control_host to the target image while retaining current worker and egress_proxy images. Meet the same settings digest and source_commit requirements. Upgrade with it, then confirm readiness and that every API instance runs the new version.

Next upgrade to the final release with target egress proxy and worker digests. This prevents proxy/workers from sending new fields before the API is ready. Apply intermediate and final test-ops inputs separately through scripts/test-ops.sh upgrade.

3. Upgrade and affected sessionsCopy Anchor Link

Run upgrade with step 1's release-new.json. A worker image change checks current checkpoints with the new image and refuses with exit 3 if sessions are incompatible. Affected ids are written to $TEST_OPS_STATE_DIR/upgrade-<timestamp>.sessions. Review them and notify users, then rerun scripts/test-ops.sh upgrade release-new.json --approve-sessions approved.txt with exactly those ids. Any difference is rejected; accepted approvals are recorded in the state directory. Recovering an incompatible session with start_fresh discards its checkpoint context. No approval is needed if none are affected.

Take a backup immediately before upgrading. Reverting means running upgrade with the previous release file; the same rules apply because the worker image changes again.

If upgrade stops before the new release starts (for example, scheduler quota check failure), the installation remains stopped with pending.json. Only status, reset, and upgrade with a release identical to pending are allowed. Other commands, including upgrade to the previous release, are rejected. Fix the cause and rerun upgrade with the same target release.

If the installation merely went down through compose down or reboot, use scripts/test-ops.sh start to reuse the same volumes, rather than upgrade/deploy (lifecycle). start rejects pending.json.

scripts/test-ops.sh upgrade release-new.json

Pass actual absolute paths outside the checkout for release-new.json and approval files. Target checkout HEAD must match release source_commit.

VerifyCopy Anchor Link

Check applied release, service health, and ready in status. Resume an existing session on a new worker; compare context, final checkpoint, and turn completion. For metric renames, update Datadog definitions/queries/filters and Prometheus rules, then verify actual series.

Compare overridden CSS names with changelog operator actions and update them too. Do not assume aliases exist for old metric/CSS names.

TroubleshootingCopy Anchor Link

With pending.json, start and upgrade to the previous release are rejected. Fix the cause and upgrade to the identical pending target. Do not reuse old approval files if the affected list changes. start_fresh discards prior context.

Roll backCopy Anchor Link

Applying a previous release still follows upgrade checks and gate rollback floors. Do not simply revert image tags to builds unaware of new schemas/checkpoints. A pending failure before the new release starts rejects the previous release; recover using the same target.

Next stepsCopy Anchor Link

Compatibility, Session recovery, Compose lifecycle.

Pre-upgrade gatesCopy Anchor Link

The first tag including both incremental checkpoints and release-tool restoration candidates is v0.1.0. That tag is the rollback floor, and the latest published release is v0.1.2. Release installations must not revert to images earlier than v0.1.0. Pre-release main-<12> installations must check this floor separately. Verify the commit and all three digests in the release manifest together.

Compare target changelog actions with your deployed version. Meet every applicable condition before enabling features or changing workers.

Test new control-host with the previous worker before deployment: complete session turn 1 → remove worker → complete turn 2 from checkpoint, with no 400 BAD_REQUEST in API worker requests. Empty-installation ready 200 does not substitute.

First applying changes after v0.1.0Copy Anchor Link

• Apply every migration from v0.1.0 through the target version (through 0262 for v0.1.2) with migrate. Old images ignore new 0239_connection_replaces_auth columns, but this alone does not make other checkpoint/protocol rollback safe.
• Compose at the v0.1.0 tag refuses to render when ANTHROPIC_API_KEY is empty. When upgrading from v0.1.0, a key left in the env file can stay. The v0.1.2 tag accepts it set or unset. Only re-applying v0.1.0 sources requires a non-empty key.
• Replace control-host before worker. Even with platform_tool_trust fallback, the transition uses old permission paths; keep the order.
• Update API, Compose maintenance, and Kubernetes checkpointGc/sessionArchive CronJobs first, confirm Jobs started from old images have finished, then update workers. Old GC can delete pinned revisions, and old APIs and verifiers reject the local branches workers now always carry. AGENT_ACTION_FLOWS and SESSION_PROMPT_REVISION no longer exist and their values are ignored. The v0.1.1 tag has no published artifacts, so do not use it; the release after v0.1.0 is v0.1.2.

Apply the conditions below after checking whether your installation already includes them. Before rollback, count affected sessions through read-only SQL. Prefer roll-forward without approving data loss where possible.

Compose checkpoint checksCopy Anchor Link

When worker images change, Compose upgrade checks calculate restoration plans for every current checkpoint using the new image's engine/SDK/CLI/profile digest. Sessions matching all four continue without approval; notify users and approve the exact INCOMPATIBLE_CHECKPOINT list. If checking cannot complete or other validation failures occur, fail closed: include every session with uncollected checkpoints and log this. Recheck after stopping writers. start_fresh discards incompatible checkpoint context. Kubernetes pre-upgrade hooks make equivalent decisions from pointer combinations without executing images (Kubernetes checks).

On a worker image change, upgrade retains the old image in the state directory's previous-worker file. For each session, the scheduler matches checkpoint engine/SDK/CLI/profile digest scheme against current/previous images, preferring current if both match (configuration). The affected list therefore includes sessions restorable by neither image. If either image check fails closed, retain the new-image list unchanged. For rebuilt images with the same combination, the scheduler selects current: do not remove sessions unverified on current merely because previous can restore them. If either is too old to report its combination (a build without combination reporting), previous is not used and the gate retains the new-image list. The support window is one release: another upgrade can affect sessions left on the older image.

Shared runtime profile preparationCopy Anchor Link

SDK upgrades also update runtime_version in example settings (tests/fixtures/settings/). Installation ceilings are saved as new versions when the built-in tool catalog changes. Profiles whose runtime_version is absent from the current image cannot accept new sessions. The exception is a version whose built-in tool catalog lists the same tools, boundaries and aliases as the new build's: the image then counts as shipping it. A profile row left at the previous version keeps starting new sessions after the upgrade, and the operator sees and does nothing. When the catalogs differ (a tool was added or changed), a new session is refused with UNSUPPORTED_CAPABILITY and a message naming the profile and version, and the operator saves a new version of that profile. Checkpoints are not judged by this: they resume only on an image with the same engine, SDK and CLI combination as the build that wrote them. Include the new engine combination's built-in tool catalog in that release too. A version lacking a catalog cannot have a deny ceiling. Newly added tools remain blocked in each new ceiling version until accepted with accept_added_tools (api-server.md). See Kubernetes rollback for its procedure and cautions.

Worker/control-host rollback and incremental checkpointsCopy Anchor Link

Workers included in v0.1.0 upload incremental bundles containing only new objects over the previous checkpoint chain. The manifest records this in workspace.baseBundles. Manifest version remains 2; only this field distinguishes incremental checkpoints. Earlier builds cannot restore checkpoints containing it because their manifest schemas reject unknown keys.

• Old worker images: manifest decoding fails with Invalid Claude checkpoint manifest: workspace Unrecognized key: "baseBundles" before clearing the workspace or downloading bundles, so no partially restored workspace remains. The fixed manifest makes relaunch fail again with worker.failed.
• Old control-host: restoration plans reject with CHECKPOINT_UNAVAILABLE without retreating to older revisions. Finalize also rejects incremental manifests and does not advance pointers. With new workers, only standalone bundles like initial checkpoints commit. Old GC skips undecodable sessions rather than deleting base bundles.
• Find affected sessions: manifest contents are not in the database. Fetch manifests using manifest_ref/manifest_version from uncollected checkpoints rows (collected_at IS NULL) referenced by sessions.checkpoint_revision, then test jq -e '.workspace.baseBundles | length > 0'. Inspect parent rows too because restoration can retreat through parent_revision. Worker worker.checkpoint.published with base_bundles above 0 indicates incremental publication.
• Before rollback: roll forward if affected sessions exist. No operator env/setting/CLI forces a new full bundle. Workers produce full bundles only when chains are absent/broken or reach length 32 (MAX_WORKSPACE_BUNDLE_CHAIN) or bundle limits. If rollback is unavoidable, affected sessions cannot restore during it; roll-forward restores them unchanged.

Control-host rollback floor: release toolsCopy Anchor Link

Control-host in v0.1.0 runs release-created sessions with release-pinned tools (agent_releases.runtime_config_snapshot.tools, within profile tools), rather than every profile tool. Claims deliver tools at worker startup; workers retain them until exit. Agents selecting tools or declaring forbidden therefore narrow even existing sessions on the next claim. Pre-deployment workers retain profile tools until WORKER_IDLE_TIMEOUT_SEC (default 1800 seconds) after their last turn without input, or until pause/restart. Sessions without releases or whose releases retain all profile tools are unchanged. A session created on a release after this deployment records sessions.runtime_tools_source = 'release', and its claim carries the release's effective_tools.tools as stored, with no intersection with profile tools. Earlier sessions read as profile, so their tools and checkpoint fingerprints are unchanged. Worker images/protocol are unchanged.

• Official release rollback floor: v0.1.0. Tool lists enter checkpoint profile digests. This release includes both profile and release tool lists as restoration candidates. Earlier pre-release builds cannot restore release-tool checkpoints and stop as recovery_required (incompatible_checkpoint). Roll forward and retry_restore the same checkpoint. Pre-release main installations should first deploy a release verified to support these candidates as a rollback target.
• Count narrowing releases before deployment using this read-only query. It also counts agents retaining every profile tool, so it is an upper bound.

WITH narrowed AS (
  SELECT r.id FROM agent_releases r JOIN agent_versions v ON v.id = r.version_id
  WHERE v.card->'capabilities'->'tools' <> 'null'::jsonb
     OR jsonb_array_length(coalesce(v.card->'capabilities'->'forbidden', '[]')) > 0
)
SELECT
  (SELECT count(*) FROM narrowed) AS releases,
  (SELECT count(*) FROM sessions s JOIN narrowed n ON n.id = s.agent_release_id
    WHERE s.admission_state NOT IN ('closing', 'closed')) AS open_sessions,
  (SELECT count(*) FROM agent_releases
    WHERE CASE WHEN jsonb_typeof(runtime_config_snapshot->'tools') = 'array'
      THEN EXISTS (SELECT 1 FROM jsonb_array_elements(runtime_config_snapshot->'tools') t
                    WHERE jsonb_typeof(t) <> 'string')
      ELSE true END) AS unreadable;

unreadable means snapshots missing tools or containing something other than a string array. Their sessions run without built-in tools (fail closed). Normal release creation always stores a string-array tools.

• Restoration candidate cap: claims allow 16 candidates (worker contract), with all profile lists before release lists. Narrowed-release sessions with many previous profiles (16 without GitHub configuration, 8 or more with it) face two effects. ① After switching to release tools, the current profile's profile list consumes another slot, displacing the oldest previous profile candidate. Checkpoints matching only that candidate stop as incompatible_checkpoint; retrying the same candidate rules cannot help. Choose start_fresh (discard context) or close. ② Rolling back to pre-release builds lacking release-tool candidate support fills the cap with profile lists, excluding release-tool candidates; release-tool checkpoints created after the switch cannot restore. Roll forward, then retry_restore to recover these.

Removing worker protocol fieldsCopy Anchor Link

Existing worker containers keep running old images after API upgrades. Even idle workers remain for WORKER_IDLE_TIMEOUT_SEC (default 1800 seconds). First make receivers accept messages without a field; remove it from senders only after every receiver is updated. Do not change both in one release.

Premature phase 2 makes old peers reject requests/responses as contract violations. Workers treat invalid heartbeat responses as nonretryable, assume lease loss at the first heartbeat, and stop.

For adding checkpoint manifest fields, readers go first. Both control-host (precommit validation, restore planning, GC) and worker (restore) reject unknown keys. Update control-host first or within the same release, then workers. Once checkpoints use the field, old workers cannot restore them for the same reason as baseBundles.

API also goes first when adding worker-sent protocol fields. Its request schemas are strict: old APIs reject entire requests containing unknown fields with 400 BAD_REQUEST.

Database schema change orderCopy Anchor Link

Migrations precede rollout, so old control-host runs briefly on new schemas and old workers remain during the support window. Keep every column/table they use until all old code is gone. Spread changes across three releases; never add and delete in one.

Renaming also removes the old name, so use the same add/migrate/switch/delete order. WORKER_IDLE_TIMEOUT_SEC is idle time after the last turn, not maximum worker lifetime. Busy workers can survive longer: decide cleanup from actual remaining containers, not elapsed time.

For example, checkpoint runtime combination columns checkpoints.runtime_* (migration 0196_checkpoint_runtime) were phase 1: nullable without defaults, leaving older rows null. 0199_worker_image_runtime (launch combinations, checkpoint digest scheme, worker_image_refusals) is also phase 1. A null checkpoint scheme reads as the first scheme (1).