Settings, keys, and ports

ContractCopy Anchor Link

WERFT_API_KEY and WERFT_BASE_URL configure SDK callers, not the server. See SDK client configuration.

This reference covers formats, requirements, and when settings apply. Read Compose upgrades and Kubernetes upgrades first for activation and rollback gates.

Installation limitsCopy Anchor Link

API and scheduler refuse startup if any of the six required values below (all except SESSION_TOKEN_LIMIT) is missing, or any value has an invalid format. They log all problems on one line using the same parser. Missing SESSION_TOKEN_LIMIT means 0 (unlimited). Apart from that 0, code has no defaults; Compose x-installation-limits provides local defaults. Running API readyz repeats the same validation as its config check.

API settingsCopy Anchor Link

At startup, API reads these through one parser. Unset values use defaults; invalid configured values (including empty strings) never silently fall back. API/scheduler/reconciler/worker numeric settings and installation limits share this rule. Missing required settings return X is required; invalid values describe the expected range, such as X must be a positive integer, without echoing the value, which could be a misplaced secret. All problems appear in one Refusing to start: API settings are invalid log line; startup stops. Running readyz repeats config validation.

Every /v1 response logs api.request.completed: Hono route_template without actual ids, method, status, duration_ms including auth/routing, operation=api_request, outcome, reason_code, and request_id in log context. Bodies/credentials/provider keys are excluded. Terminate also logs Terminate control transaction completed under the same request_id, with millisecond pool checkout, binding/launch/session locks, individual SQL, commit/rollback, and total control-transaction timing. Join the logs by request_id to diagnose acceptance latency.

Immediate terminate of an already absent execution, or scheduler/reconciler confirmation of absence, emits control.result with operation=terminate, receipt control_id, outcome, reason_code, and effect_ms from durable acceptance to confirmation (equal to duration_ms). Overdue receipts first emit outcome=unknown/reason_code=deadline_exceeded; later confirmed absence upgrades the receipt and adds outcome=succeeded/reason_code=execution_gone.

WEB_DIST_DIR enables same-origin console serving. GET/HEAD outside /v1 and /internal serve files or index.html SPA fallback, allowing /login and /bootstrap refreshes. No fallback applies under /v1 or /internal; authenticated /v1/nope returns JSON 404 NOT_FOUND. Hashed /assets/ files cache immutable; missing assets return 404, not a page. index.html uses no-cache and CSP script-src 'self', prohibiting inline scripts. control-host's web build stage builds apps/web and retains only dist; no separate web image exists. For local web-only work, bun run --cwd apps/web dev uses Vite and proxies /v1 to VITE_API_TARGET (default http://127.0.0.1:3000).

AUTH_MODE=none cannot coexist with EGRESS_AUTHORIZER_PORT. An enabled authorizer means workers reach API through proxy; none would let worker code impersonate arbitrary owners via X-Owner-Id, so startup refuses. Readiness validates AUTH_MODE itself (none|api-key).

Workers also cross-check startup settings. Missing WORKER_PROVIDER_MAX_RETRIES fails: this installation limit has no code default and scheduler always passes it. WORKER_NEXT_INPUT_WAIT_SEC must be below WORKER_REQUEST_TIMEOUT_SEC or every long poll times out. Scheduler/reconciler treat empty DATABASE_URL as unset and fall back to QUEUE_DATABASE_URL.

API/slack read current/previous runtime combinations through WORKER_IMAGE_RUNTIME/WORKER_IMAGE_PREVIOUS_RUNTIME. Copy runtime from images.yml worker digest JSON rather than typing it ({"engine":…,"sdkVersion":…,"cliVersion":…,"profileDigestScheme":…}). The engine is claude or codex; other names fail rendering with images.worker.runtime.engine must be one of: claude, codex. Custom images lack that JSON; extract with the same command:

docker run --rm -i --network none --entrypoint bun <worker-image> \
  run apps/worker/src/image-runtime.ts <<<'{}' | jq -c 'del(.profiles)'

One worker image holds one engine. apps/worker/Dockerfile picks it with the build argument WORKER_RUNTIME. Empty builds the Claude image; --build-arg WORKER_RUNTIME=codex_app_server builds the Codex image (the build log prints codex-cli 0.167.0, and any other version fails the build). CI builds the Codex image as its own image, werft-worker-codex. A deployment that runs Codex points WORKER_IMAGE (the chart's images.worker) at that image and sets the runtime read with the command above (engine is codex). The local stack builds the same image with the infra/compose.codex-worker.yml overlay. A deployment whose Codex sessions call the real OpenAI Responses API also adds api.openai.com:443 to the proxy's EGRESS_CREDENTIAL_ALLOWLIST. The Codex engine has no budget of its own. The session cost and token limits (SESSION_COST_LIMIT_USD, SESSION_TOKEN_LIMIT) are enforced by the worker gateway on every provider call from the stored usage, and a session past a limit gets 403 BUDGET_EXCEEDED on its next call. Limit of this: usage reaches the gateway through the proxy's report. A report lost after its retries (1s, 5s, 15s) leaves the stored cost low, and that session's later provider calls are then stopped only by the proxy's in-process gate (one process, lost on restart). The limit is therefore a soft one that relies on the stored usage. After a lost report, with several proxies or a restart, the overshoot has no bound. Even with every report delivered, one long response, parallel calls, and a call sent while a report is still in flight can pass the limit (the Claude engine's budget is also checked only after a call completes). Closing this with durable usage records is follow-up work.

Switching engines. One deployment cannot pick between two engines' images at once: new sessions always start on the current image, and the previous image only resumes checkpoints. Choosing the engine at install time is the simplest path. To replace the WORKER_IMAGE of a running deployment with another engine's image, follow the upgrade check (upgrade-check) as it is: the new configuration's WORKER_IMAGE_PREVIOUS must be the image deployed now, and every session the check reports as unresumable (not only a different checkpoint engine, but also a mismatched SDK or CLI version or digest scheme, or an unrecorded runtime) must be approved in upgrade.approvedSessions as exactly the list the check prints. Checkpoints do not move between engines, so the previous (old-engine) image can only be emptied once the sessions resuming on it have ended.

With neither WORKER_IMAGE_RUNTIME nor WORKER_IMAGE_PREVIOUS_RUNTIME, no runtime reading or pin preservation applies. Previous-only or unreadable values reject startup. When configured, open sessions keep pins created with the previous image even after profiles advance runtime_version. Profiles absent from current-image runtime cannot accept new sessions. A previous version whose built-in tool catalog matches the current image's counts as shipped, so that profile keeps starting new sessions after the upgrade. SDK upgrades with a different catalog therefore create new ceiling versions with updated runtime_version (console or POST /v1/installation-profiles).

Set the following only to send GitHub Packages bot tokens (decision) to another host, such as GHES/test registries. URLs must be http(s) without userinfo/query/fragment, or startup fails. Hosts must also be in proxy EGRESS_CREDENTIAL_ALLOWLIST, or EGRESS_CREDENTIAL_PRIVATE_ALLOWLIST for private addresses.

Scheduler, worker container, and egress settingsCopy Anchor Link

Scheduler reads these to create workers. Unset values use defaults; invalid formats reject startup. User, HOME/workspace paths, tmpfs sizes, credential port, and stop timeout enter werft.isolation (isolation); changing them replaces running workers. CPU/memory/PID limits apply next launch.

WORKER_MEMORY_MB is chosen with MAX_TRANSCRIPT_BYTES; see capacity for measurements, node counts, and changes.

Scheduler receives current WORKER_IMAGE/WORKER_IMAGE_RUNTIME and previous WORKER_IMAGE_PREVIOUS/WORKER_IMAGE_PREVIOUS_RUNTIME, using API-equivalent runtimes. Previous image/runtime must be set together; one alone fails. Without current runtime every launch uses current image. Otherwise, match each checkpoint engine/SDK/CLI/profile digest scheme to current/previous, preferring current when both match. New sessions, checkpoint-free sessions, and old unrecorded checkpoints use current. No match stops before launch as recovery_required (incompatible_checkpoint), with checkpoint_restore_failed detected_at: "launch". retry_restore is available.

Workers report actual image runtime at claim. Mismatch causes API LAUNCH_FAILED with reason_code worker_image_mismatch and records refusal for that image/configuration. Further launches for it wait; scheduler logs images_refused as error and exits 1. Correcting image configuration (changed values) restores launches next pass. Sessions themselves do not become failed.

Deploy API first, then scheduler WORKER_IMAGE: new workers send claim image_runtime, rejected by old APIs.

EXECUTION_DOCKER_NETWORK/EXECUTION_DOCKER_NETWORK_ALLOWLIST are no longer read; configured values reject startup.

Proxy and API authorizer use these settings. Lists replace, rather than extend, defaults: restate required providers/registries when adding GitHub/bucket hosts. Public credential lists must match in proxy/API. Private lists enforce proxy rejection but only console warnings in API. Matching private lists is recommended, not required; core/local-model role defaults also differ.

Compose's three ports/two internal URLs come from x-egress-ports at the top of infra/compose.core.yml. Proxy EGRESS_PROXY_PORT/EGRESS_CREDENTIAL_PORT/EGRESS_AUTHORIZER_URL, API EGRESS_AUTHORIZER_PORT, and scheduler EXECUTION_EGRESS_PROXY_URL/EXECUTION_EGRESS_CREDENTIAL_PORT all use it. Change listener/URL pairs together. A test compares both URL ports with all three listeners in docker compose config.

API memory is determined by checkpoint bundle validation: at most two validations concurrently, each running git fetch and index-pack together under individual CHECKPOINT_GIT_MEMORY_MB address-space limits. Change both settings together; raising only the limit leaves OOM victim selection to the kernel. See the api service comment in infra/compose.core.yml.

Compose Postgres uses POSTGRES_USER (postgres), POSTGRES_PASSWORD (dev), POSTGRES_DB (sessions); all services build DSNs from the same values.

Workers also read the following env settings, but scheduler does not forward them, so production uses code defaults: WORKER_ANSWER_POLL_SEC (1), WORKER_ATTACH_FETCH_TIMEOUT_SEC (300, fetch one repository attached mid-session), WORKER_ATTACH_FETCH_MAX_MB (2048, fetched scratch mirror size), WORKER_CLAIM_TIMEOUT_SEC (60), WORKER_DRAIN_TIMEOUT_SEC (100, reduced to fit stop grace), WORKER_HEARTBEAT_INTERVAL_SEC (5), WORKER_IDLE_TIMEOUT_SEC (1800), WORKER_LEASE_SAFETY_MARGIN_SEC (10), WORKER_NEXT_INPUT_RETRY_SEC (60), WORKER_NEXT_INPUT_WAIT_SEC (20), WORKER_REQUEST_TIMEOUT_SEC (30), WORKER_STARTUP_TIMEOUT_SEC (3600), QUESTION_TIMEOUT_SEC (1800), WORKER_CLAUDE_CONFIG_DIR ($HOME/.claude).

Supervisor loopsCopy Anchor Link

Compose does not forward RECONCILER_STATUS_FILE, SCHEDULER_STATUS_FILE, or RECONCILER_DRY_RUN. Use status files/--health; run reconciler dry-run directly once through recovery. Only API reads HEARTBEAT_TTL_SEC; reconciler rejects it.

Kubernetes env and valuesCopy Anchor Link

EXECUTION_BACKEND selects local_docker (default, also when empty) or kubernetes; other values reject startup with an error naming the value. Kubernetes ignores Docker settings (DOCKER_HOST, EXECUTION_DOCKER_*, EXECUTION_WORKSPACE_QUOTA) and socket, and skips Docker egress-isolation/quota preflights. Chart NetworkPolicy provides isolation; the workspace requests PVC size, while actual enforcement depends on storage. Fixed stop/request timeouts are 120s + 30s, requiring SCHEDULER_PASS_TIMEOUT_SEC >150. migrate-workspace is local_docker-only.

On readOnlyRootFilesystem Pods, status files need writable directories. Default /tmp/scheduler-status.json and /tmp/reconciler-status.json work with /tmp emptyDir. Override paths with SCHEDULER_STATUS_FILE/RECONCILER_STATUS_FILE. Loops write adjacent <status file>.<pid>.tmp then rename; scheduler also stores .quota-verified nearby. The whole directory, not just the file, must be writable.

Shared env matches Docker: WORKER_IMAGE, WORKER_GATEWAY_URL, EXECUTION_EGRESS_PROXY_URL, EXECUTION_EGRESS_CREDENTIAL_PORT, EXECUTION_INSTALLATION_ID, EXECUTION_WORKSPACE_QUOTA_MB, EXECUTION_WORKSPACE_GC_MIN_AGE_SEC, S3_BUCKET, AWS_REGION. Kubernetes adds constraints:

• WORKER_IMAGE must be repository@sha256:<digest>.
• EXECUTION_INSTALLATION_ID must be a lowercase DNS label, at most 32 characters.

Kubernetes-only settings:

EXECUTION_WORKSPACE_ENFORCED_BY_STORAGE defaults to false when unset and accepts only exact true/false. Empty values, whitespace, uppercase, 1/0, and on/off reject startup without exposing the original value. Helm generates it from worker.workspace.enforcedByStorage (default false): an operator declaration that storage hard-enforces requested bytes for every workspace PVC in this installation, including existing PVCs. With an empty StorageClass, the cluster default applies and operators must verify enforcement for each PVC. Werft does not verify it. Only the declaration path passing the heuristics of a dedicated writable mount (not /, rw present, ro absent), filesystem size 0.95–1.05 times the request, and PVC capacity at least the request yields bytes hard/inodes unknown. See Storage and limits. Changing the declaration leaves PVCs, the isolation stamp, and existing Pod env unchanged; only new Pods receive the new value.

Scheduler logs failed-worker termination reasons and last 8 KiB under Execution resource failed; log tail kept. See storage/retention in the repository's Kubernetes execution adapter README.

Chart-generated valuesCopy Anchor Link

With deploy/helm/werft (installation), chart values generate these env settings:

Any Deployment accepts extraEnv additions. Secret-sourced DATABASE_URL, EGRESS_AUTHORIZER_TOKEN, AWS keys, and catalog credentials are listed in installation prerequisites.

ProbesCopy Anchor Link

healthz checks HTTP response only, not database. Use it for liveness; readyz checks DB/schema/config for readiness only. Temporary DB loss removes API traffic without restarting Pods. No /livez exists. Scheduler/reconciler/slack health reads loop status files and fails after last-pass failure, current-pass timeout, or no completed pass within SCHEDULER_HEALTH_STALE_SEC/RECONCILER_HEALTH_STALE_SEC/SLACK_ROLE_HEALTH_STALE_SEC. Startup waits for the first pass.

Chart resourcesCopy Anchor Link

Secrets and AWS valuesCopy Anchor Link

For static S3 credentials, set secrets.objectStoreKeys: true and supply both AWS keys. With both present, static keys precede IRSA.

Usage and cost meteringCopy Anchor Link

• Credential routes meter session cost. Proxy reads usage from 2xx /v1/messages responses, JSON or SSE, and reports POST /usage to authorizer when responses finish/disconnect. count_tokens/error responses do not count.
  • Incomplete usage is estimated conservatively. Streams ending before message_stop count each delivered content character as one output token and each delivered web-search result block as one search. Unreadable JSON (truncated, over 8 MiB, or invalid JSON) uses request estimation: 1 request byte/input token and full max_tokens/output. Proxy logs Provider usage estimated; ledger estimated=true.
  • API converts through prices to USD, rounds up to micro-dollars, writes one provider_usage row/exchange and adds session cost in the same transaction, keeping ledger/session totals aligned.
  • Engine calls and tool calls made directly with engine tokens are metered equally.
  • Proxy assigns exchange ids. Failed authorizer reports retry after 1, 5, and 15 seconds; ids count once. Final failure logs Provider usage went unreported and leaves that exchange uncounted.
• SDK turn-end total_cost_usd increments appear only in turn result cost_usd, not session cost. Engine maxBudgetUsd still uses SDK values. Pre-proxy-metering sessions retain costs previously accumulated from SDK values.
• Platform code owns prices by model: input/output/cache write (5-minute 1.25×, 1-hour 2×)/cache read. New models/prices require release updates.
  • Fast prices currently cover Opus 5.5 ($8/$40) and Opus 5/Opus 4.8 ($10/$50). Cache multipliers apply to fast prices. speed uses response usage.speed, then request speed, then standard.
  • Unknown models, fast responses without fast prices, or unknown speed values use per-item highest rates across both tables. Authorizer warns Provider usage priced at the fallback rate; ledger priced_by=fallback.
  • Response usage.server_tool_use web searches add $0.01 each ($10/1,000). Web fetch has no charge beyond tokens. Code execution is excluded: free with web search/fetch, otherwise billed by container time with organizational free allowances, not invocation count. All three counts and speed are recorded in provider_usage.
  • Models from 4.6 onward apply 1.1× to every token price (fast/cache included) for inference_geo: "us", excluding web-search fees. Geo follows response usage.inference_geo, then request inference_geo, then unknown (workspace may default to us). Non-global/us values use highest prices ×1.1, priced_by=fallback. Ledger records inference_geo. Old APIs reject this new field with 400, causing unreported exchanges. Update API including migration before proxy (Compose order, Kubernetes order). New API accepts old reports without it.
  • Pre-4.6 models (Opus 4.5/4, Sonnet 4.5/4, Haiku 4.5) do not support inference_geo and use standard prices regardless of geo. Their window is 200K without published long-context rates; input+cache write+cache read over 200K uses highest prices, priced_by=fallback. Models 4.6 onward use standard prices for the full 1M.
  • Batch is irrelevant: credential routes do not expose /v1/messages/batches.
• Token limits sum input_tokens + cache_creation_input_tokens + cache_read_input_tokens + output_tokens from each provider_usage row into sessions.provider_tokens through an INSERT trigger in the same transaction. This includes calls from pre-0127 APIs during rollout/rollback. Cache reads count as model-processed tokens; estimates count as estimated. The migration backfills provider_usage totals, but cannot recover tokens from older SDK-cost-only sessions. Totals saturate at 2^53-1 (Number.MAX_SAFE_INTEGER, migration 0129). Numeric trigger arithmetic prevents oversized tokens from rolling back usage/cost writes.
  • SDK has no token-budget option; in-turn enforcement relies on egress rejection/reauthorization below. Calls opened after reaching limits are rejected and turns finish failed(budget_exceeded); later turns are blocked with BUDGET_EXCEEDED. Proxy uses Messages error format with message starting BUDGET_EXCEEDED:, but does not determine reason from text. Engine reports the 403 as api_error; exchanges cut by reauthorization also remain api_error if retries are exhausted (PROVIDER_MAX_RETRIES=0). Finalize converts api_error to budget_exceeded and receipt BUDGET_EXCEEDED if the session is already over budget, clearing provider_error diagnostics. A 403 below limits remains api_error.
• Cost/token limits are judged after calls finish. Concurrent open calls may all pass authorization; active turns can exceed limits.
• Authorizer rejects provider egress tokens at or above cumulative cost/token limits with 403 BUDGET_EXCEEDED. New exchanges fail immediately; open exchanges disconnect at the next reauthorization (every 30 seconds). Repository/object-store routes remain available.

Metrics roleCopy Anchor Link