Upgrades and backups
Three things carry versions: the server image, the CLI, and the actions. Postgres is the only state to back up.
What is versioned
| Component | Where | Versions |
|---|---|---|
| Server | ghcr.io/stackorder/stackorder | X.Y.Z and latest, from tags vX.Y.Z on stackorder/stackorder |
| CLI | Release assets on stackorder/stackorder | stackorder_X.Y.Z_<os>_<arch>.tar.gz with a SHA-256 checksum file |
| Actions and reusable workflows | stackorder/actions | v1 and v1.x.y; the v1 tag moves independently of server releases |
Workflows reference the reusable workflows at @v1. The stackorder-version input pins the CLI release that setup installs; it defaults to latest.
The API between the CLI and the server is v1. Fields are only ever added and both sides ignore fields they do not know, so the server and the CLI can be upgraded independently within v1.
Upgrading the server
- Read the release notes for the new version.
- Take a database snapshot if the release includes migrations.
- Deploy the new image tag.
- The new instance runs pending migrations at start-up, under a migration lock, so only one instance applies them.
- The previous instance keeps serving until the new one passes
/readyz.
On AWS with the Terraform module, this is a change of the image tag and an apply. See Deploy on AWS.
To apply the migrations as a separate step before the server rolls — so a schema change lands under your control, not as a side effect of the first new instance starting — run the new image's migrate command against the same database first:
docker run --rm -e DATABASE_URL="$DATABASE_URL" \
ghcr.io/stackorder/stackorder:X.Y.Z migrateIt takes the same migration lock as start-up, so it is safe while the old instances keep serving, and stackorder-server migrate --status reports whether a database is already at the image's version (exit 0), has migrations pending (exit 2) or is dirty or ahead of the image's version (exit 1). This is optional; without it the new instance migrates at start-up as before. See Migrating.
If a migration fails — whether at start-up or from migrate — it leaves the schema marked dirty, and every later start and migrate refuses the database until the flag is cleared, so the backup from step 2 is what you roll back to. Each migration runs in one transaction, so a failed one rolls back and leaves the schema at the previous version. To clear the flag after checking or repairing the schema, use stackorder-server migrate --force N with the version the schema is at; see Recover from a dirty migration for the full procedure.
In-flight runs survive an upgrade. Work is queued in Postgres and every handler is idempotent: on SIGTERM an instance hands unstarted work back to the queue and gives running handlers 30 s, and anything a killed instance had claimed is claimed again after 10 minutes. The per-minute reconciliation catches any wave that finished during the switch. GitHub does not retry a failed webhook delivery by itself; redeliver it from the App's delivery log if needed.
Upgrading from v0.5.0 crosses a window where old and new replicas share the database. A v0.5.0 replica clears a stack's removed_at on every plan save; a v0.6.0 replica clears it only when a merge moves the default-branch graph. So while a v0.5.0 replica is still serving, an open pull request it replans can revive a stack that a merged pull request already deleted from the default branch, and because that stack is no longer on the default branch no later merge re-marks it — it stays in the stack lists and the overview count until its directory returns to the default branch. Complete the rollout promptly to keep the window short; the effect is cosmetic (a deleted stack counted as live) and touches no run or state.
The server applies an installation's suspend and unsuspend webhooks in receive order (see the data model). This guarantee holds once every replica runs v0.6.0 or later. While a v0.5.0 replica and a new one run together during the rollover, a suspend or unsuspend the old replica processes still takes effect in the order it is processed, as v0.5.0 always did, not by receive time; the daily installation sync and the finished upgrade settle it, so no action is required. To avoid even that transient, drain the old replicas before the new ones serve.
API key scopes are enforced only by a replica on v0.6.0 or later. Migration 0016_api_key_scopes stores a scoped or expiring key in the same table, and under the same SHA-256 hash, that v0.5.0 accepts, and a v0.5.0 replica reads neither the scope columns nor the expiry: it authenticates every unrevoked key as a full, installation-wide, never-expiring administrator. So during the rolling upgrade a read-only, installation-scoped or expired key minted by a new replica is honoured as an unrestricted administrator, including past its expiry, by any v0.5.0 replica still serving. Existing keys are unaffected — they are unscoped administrators already — so mint and hand out scoped or expiring keys only once every replica runs v0.6.0 or later. Before rolling back to a version without scope enforcement, revoke every scoped or expiring key you issued, because that version ignores all three restrictions (see Rolling back).
During a rolling deploy already-running old replicas keep serving and draining normally, but a replica of the previous version cannot start once the new one has migrated the database: its start-up migration refuses a schema newer than it knows. So do not count on restarting or rescheduling an old replica after the migration has run without first rolling the schema back or restoring the snapshot.
To go back to a previous release, see Rolling back.
Rolling back
The server only migrates up: an older image refuses to start when the schema is at a version it does not embed. So before you redeploy a previous image, revert the schema to that release's version with the newer image's migrate --down — it embeds the down migrations the older one does not:
Read the table below for the schema version of the release you are going back to, and what reverting to it drops.
If the rollback reverts migration 0016 — any target below schema 16 — revoke every read-only, installation-scoped or expiring API key first. Reverting 0016 drops the scope columns, so the older schema and release honour every surviving key as a full, never-expiring administrator (see the version 15 row below). List the keys and their scopes with
stackorder-server api-key listand revoke each restricted one withstackorder-server api-key revoke <id>.Take a database snapshot if the downgrade drops data you want to keep.
Stop every instance of the newer release, or scale it to zero. A newer instance left running fails once the columns and tables it uses are gone, and keeps writing data the older schema has no place for.
Preview the plan, then apply it, with the image you are rolling back from:
shdocker run --rm -e DATABASE_URL="$DATABASE_URL" \ ghcr.io/stackorder/stackorder:X.Y.Z migrate --down --to N docker run --rm -e DATABASE_URL="$DATABASE_URL" \ ghcr.io/stackorder/stackorder:X.Y.Z migrate --down --to N --yesWithout
--yesit lists the migrations it would revert and the data each loses and changes nothing;--yesreverts. It takes the same migration lock as start-up, so it is safe beside a starting server, but not beside a running one — stop them in step 4.Deploy the previous image. It starts because the schema is now at or below the version it embeds, and does not migrate.
A downgrade is purely additive to undo only when every migration it reverts is additive; where it drops data, that data is gone unless you restored a snapshot. Reverting below the oldest release's version is a teardown, not a rollback, and migrate --down refuses --to 0.
| Release | Schema version |
|---|---|
| v0.1.0, v0.2.0 | 6 |
| v0.3.0 | 7 |
| v0.4.0 | 8 |
| v0.5.0 | 9 |
| v0.6.0 | 16 |
Reverting down from version 16 drops, one version at a time:
| To version | Reverting drops |
|---|---|
| 15 | The API-key scopes, and with them every restriction on every key. api_keys.read_only, api_keys.installation_id and api_keys.expires_at are dropped, so once the schema is below 16 — and under the older release, which never read these columns — every read-only, installation-scoped or expiring key is honoured as a full, installation-wide, never-expiring administrator, including past its expiry. Revoke every scoped or expiring key before you run migrate --down below 16: list the keys and their scopes with stackorder-server api-key list and revoke each restricted one with stackorder-server api-key revoke <id>. Unscoped keys are unaffected — they were full administrators already. |
| 14 | The trace context carried across the work queue (events.trace_context and jobs.trace_context); queued and in-flight rows lose the stored W3C trace context, so a worker span no longer continues the trace of the request or job that enqueued the row. Observability only — no run or state data is affected. |
| 13 | The observation time that orders each installation's suspend/unsuspend webhooks in receive order (installations.suspend_observed_at); the older release resolves suspend and unsuspend by processing order again, as v0.5.0 did, and the daily installation sync settles each installation's suspension on its next run. |
| 12 | Nothing: the run_stacks_stranded_idx index that finds stranded plan objects to reclaim; the next upgrade re-creates it. |
| 11 | The object_deletions queue and every row in it: objects already unreferenced in Postgres but not yet deleted from the artifact bucket are then never deleted and stay orphaned there, so reclaim their space out of band (for example with a bucket lifecycle rule) if you need it. |
| 10 | Nothing is dropped, but the repos.default_graph_id foreign key becomes NOT DEFERRABLE again, so the single-transaction data-only restore no longer works; restore a full snapshot instead. |
| 9 | Nothing is dropped; the one-time stacks.removed_at reconcile is simply not re-run. |
| 8 | Every stack's resource list (stack_resources); each stack's next apply or drift job records it again. |
| 7 | Where each stack's state object lives (stacks.state_object); until each producer's next apply or drift job records it again, stacks that read shared outputs fail to plan. |
| 6 | The comment id tying a run to its pull-request comment (runs.comment_id); the next run opens a new comment instead of editing the old one. |
| 5 | The node keys a change reached each stack through (run_stacks.via). |
Going below version 5 drops the run-service columns, then the authentication, queue, run and graph tables in turn — a full teardown rather than a release downgrade.
Upgrading the CLI and actions
- Set
stackorder-versionin the calling workflows to pin the CLI, and bump it to move. Left atlatest, every run installs the newest release. - The
v1tag ofstackorder/actionsmoves on its own. Callingplan.yml@v1.x.yinstead of@v1pins the workflow file, but the reusable workflows still call the actions at@v1. - If the server sets
STACKORDER_REQUIRED_WORKFLOW_REFor the AWS roles pinjob_workflow_ref, keep the pinned pattern in step with the tags you use.refs/tags/v1*covers everyv1release. - CLI v0.5.0 and later report each stack's resource list from apply and drift jobs. An older server ignores it, and a stack's page shows no list until a job on such a release has reported one.
Backups
Postgres holds the graphs, the run history, the locks, drift results, sessions, API keys, the work queue, where each stack's state object lives and each stack's resource list, with no attribute values. Terraform state itself is never there.
- RDS or Aurora: turn on automated backups with point-in-time recovery.
- Elsewhere: use your platform's snapshots or
pg_dumpon a schedule.
Data-only dumps
A snapshot or a plain pg_dump restores the whole database, schema included. If instead you keep a data-only dump (pg_dump --data-only) to load into a database the server has already migrated, dump everything but the migration bookkeeping and restore it in a single transaction:
pg_dump --data-only --no-owner --no-privileges \
--exclude-table-data=schema_migrations "$SOURCE_URL" -f stackorder-data.sql
psql --single-transaction -v ON_ERROR_STOP=1 "$TARGET_URL" -f stackorder-data.sqlrepos and graphs reference each other, so no load order satisfies both foreign keys row by row. repos.default_graph_id is DEFERRABLE INITIALLY DEFERRED, so the cycle is checked once at commit instead: the whole dump must load in one transaction, which --single-transaction (or pg_restore --single-transaction for a custom-format dump) does. This needs no superuser, so it works on a managed Postgres where pg_restore --disable-triggers is not available.
Restore into a database migrated to the same version the dump was taken at. --exclude-table-data=schema_migrations leaves out the migration bookkeeping row the target already has; without it the load fails on a duplicate key in schema_migrations.
What a lost database costs you: history, drift results, locks, and the state locations shared outputs read from and the resource lists, which each stack's next apply or drift job records again. Plans and applies keep working against your S3 state once a new database is in place, because nothing in the server is needed to operate Terraform, except by stacks that read shared outputs: their plans fail until each producer's next apply or drift job records its state location again.
Restoring
- Stop the server, or scale it to zero.
- Restore the database to the chosen point.
- Start the server. It runs any migrations the restored schema is missing.
- Check
GET /v1/overviewfor locks held, and compare them with open pull requests. A lock restored from before an apply finished, or a lock lost in the restore, needs a human decision: re-apply from the pull request, or release it withstackorder unlock. - Push to open pull requests, or comment
stackorder plan, to re-plan anything whose results were lost.
Rotating secrets
| Secret | How to rotate |
|---|---|
| App private key | Generate a new key in the App settings, update GITHUB_APP_PRIVATE_KEY, redeploy, then delete the old key. Runners never see it. |
| Webhook secret | Change it in the App settings and in GITHUB_WEBHOOK_SECRET together; deliveries in between fail signature checks and can be redelivered from the App's delivery log. |
| OAuth client secret | Generate a new one in the App settings, update GITHUB_OAUTH_CLIENT_SECRET, redeploy. |
| Session key | Change STACKORDER_SESSION_KEY on every instance at once. Everyone is signed out. |
| API keys | Only their SHA-256 is stored. Create a new key, switch the automation to it, then revoke the old one; see API keys. |
With the Terraform module, a new value in one of its ephemeral inputs reaches the App secret only in an apply that also increases secrets_version, and db_password_version rotates the database password. Each increase of secrets_version also generates a new session key and metrics token unless session_key and metrics_token are set: everyone signs in again, and scrapers must read the new token.
There are no runner-side secrets to rotate: runners authenticate with short-lived OIDC tokens.