Back up and rehearse restoration

What you will doCopy Anchor Link

Back up a Compose installation's database and checkpoint objects, then compare pointer versions in a separate project with a new, empty bucket. Record API verification and actual session resumption as separate evidence.

Before you beginCopy Anchor Link

You need a healthy installation, separately stored copies of the current release and secret keys, enough backup space, and a maintenance window to stop writers. Backups include the entire database and API key hashes: restrict access to 0700 and encrypt them in transit and at rest. Do not restore into the production bucket. Release bundles reject backup, reseed, and restore-drill; run them from a checkout of the installation's commit.

StepsCopy Anchor Link

1. Back upCopy Anchor Link

scripts/test-ops.sh backup

The command stops api, scheduler, reconciler, and workers, bundles the database and objects into backups/backup-<timestamp> under the state directory, then restarts services. Check for active turns before starting: they are interrupted and can later be reclaimed. Never use failed (.failed) or incomplete (.partial) backups as successful ones. The command fails if it cannot restart services, even if the backup itself finished.

Run this daily during a quiet period and immediately before every upgrade. An example retention policy is 14 days on the host and 90 days in an encrypted external store. Alert if no new successful backup exists for 26 hours, the command exits non-zero, or services fail to resume.

2. Restore to a separate destinationCopy Anchor Link

For S3, prepare a new, empty drill bucket every time. The command rejects the source bucket and any destination containing object versions or delete markers.

# Restore a backup into a throwaway project and a new, empty bucket, then verify it
scripts/test-ops.sh restore-drill /var/lib/werft/test-ops/backups/backup-<timestamp> --bucket <drill-bucket> --port-base 25432

--port-base sets the drill services' separate starting port. Choose a value that does not overlap other installations. Omit --bucket for a LocalStack trial. The drill restores postgres and the object store, then compares every checkpoint pointer by version. It does not start the API. Run it monthly and on the first backup after each upgrade.

3. Checkpoint GCCopy Anchor Link

Schedule this weekly, without overlapping backups, and start with a dry run.

API_CONTAINER='<api-container-id-from-this-installation>'
docker exec -e CHECKPOINT_GC_DRY_RUN=1 "$API_CONTAINER" \
  bun run apps/control-host/src/api/checkpoint-gc.ts

Identify API_CONTAINER from this installation's status and project label. Review the objects to retain and delete in the output, then run without the dry-run environment variable. Do not select another installation in the default Compose project. Kubernetes uses the chart's checkpointGc.schedule.

VerifyCopy Anchor Link

Check the backup exit code and status. Keep the drill logs with the actual checkpoint count, version count, and checksum results. nothing was compared means the comparison was empty; it does not prove data restoration. The previous LocalStack run had 0 objects and 0 checkpoint rows. Restoration from AWS S3 and actual session resumption with a running API have not been verified. Verify that sessions resume with the restored API and workers in a separate, isolated environment.

TroubleshootingCopy Anchor Link

Stop if the drill destination is the source or is not empty. If legal holds or retention prevent version deletion, investigate the destination and retention contract instead of removing protection arbitrarily. Drill output is recorded under restore-drills/ and in history.

Roll backCopy Anchor Link

The drill tool cleans up its project when it exits. If you used --keep for investigation, bring down only that project after checking it, and handle volumes under a separate retention policy. Keep the drill bucket distinct from production. After confirming it contains no data to reuse, release each object version's legal hold, delete versions and delete markers, then delete the empty bucket. Wait for any remaining retention period to expire; do not bypass it. Do not delete production backups for a test.

Next stepsCopy Anchor Link

Upgrade Compose, Storage and limits. For Kubernetes database and S3 restoration, follow the operator backup contract.