Changelog
All notable changes to this repository are documented here. The format follows Keep a Changelog, and releases follow Semantic Versioning. The CLI, the server image and the Terraform module are released together from one vX.Y.Z tag.
Unreleased
0.6.0 - 2026-11-15
Upgrading from v0.5.0. Server start applies seven new migrations automatically, under the migration lock, and leaves the schema at version 16; take a database snapshot first, as for any release with migrations. None needs a superuser. Once any of them is applied, a v0.5.0 server refuses to start against the database, so a drained v0.5.0 replica cannot be restarted, and going back means reverting the schema with this release's stackorder-server migrate --down --to 9 --yes or restoring the snapshot (see Rolling back).
0010_reconcile_removed_stacksmarks removed every stack absent from its repository's current default-branch graph, unless a newer graph still lists it. It changes no schema.0011_deferrable_default_graph_fkmakes the foreign key ofrepos.default_graph_idDEFERRABLE INITIALLY DEFERRED, for data-only restores.0012_object_deletionsadds theobject_deletionsqueue of plan objects to delete from the artifact bucket.0013_reclaim_stranded_plan_objectsadds the partial indexrun_stacks_stranded_idxwith a plainCREATE INDEX, which blocks writes torun_stackswhile it builds.0014_installation_event_orderaddsinstallations.suspend_observed_at, which orderssuspendandunsuspendby receive time.0015_queue_trace_contextaddsevents.trace_contextandjobs.trace_context, the trace context carried across the queue.0016_api_key_scopesaddsapi_keys.read_only(defaultfalse),api_keys.installation_idandapi_keys.expires_at.
During a rolling upgrade, v0.5.0 and v0.6.0 replicas share the database until the last old one stops:
suspendandunsuspendare applied in receive order only once every replica runs v0.6.0; until then a v0.5.0 replica applies them in processing order, which the daily installation sync settles.- A v0.5.0 replica still clears
stacks.removed_aton every plan save, so it can revive a stack already deleted from the default branch; complete the rollout promptly. - API key scopes and expiry are enforced only by v0.6.0 replicas: a v0.5.0 replica honours a read-only, installation-scoped or expiring key as a full administrator that never expires. Create such keys only once every replica runs v0.6.0, and revoke them before reverting the schema below version 16, because the older schema has no scope columns and treats every remaining key as a full administrator.
- The sample
StackorderGitHubRateLimitLowalert matchesinstallation!="", so the unlabelled series a v0.5.0 replica exports at0does not trip it.
The server and the CLI can be upgraded in either order. The one exception is a plan workflow triggered on a push to the default branch: on such a push a v0.6.0 CLI records the graph through an endpoint a v0.5.0 server does not have, so upgrade the server first. No change in this release needs a newer stackorder/actions; no stackorder/actions release triggers the plan workflow on a push yet (v1.2.0 does not), so the new endpoint stays unused until one does.
Behaviour changes for an existing deployment:
- A failing
post-applyhook now fails its stack: its dependents are blocked and not dispatched, the run fails, and abefore_mergeapply keeps its orchestration locks held. - A GitHub call that creates a check run, a comment, a drift issue or a
workflow_dispatchis no longer retried once the request may have reached GitHub; such a failure now surfaces as an error the server recovers from rather than as a silent duplicate. - With an artifact bucket, a stack's plan objects are deleted when
STACKORDER_PLAN_TEXT_RETENTION(default720h) prunes its plan text and when its repository is removed, which needss3:DeleteObjecton the server's role; the Terraform module already grants it. Without it nothing fails, but the objects stay andstackorder_pending_object_deletionsstays above zero. stackorder_github_rate_limit_remainingcarries aninstallationlabel (an installation id, orapp) and is absent until the first GitHub response; dashboards and alerts must select a label.- The installation sync forgets an installation GitHub's listing omits only after
GET /app/installations/{id}answers404, and a degraded or short listing deletes nothing. - Stacks deleted from the default branch are marked removed and leave the stack lists and the overview counts; migration 0010 does this for stacks deleted before the upgrade.
- A merged pull request records its graph as the default-branch graph only while its merge commit is still the branch head, so where automation pushes another commit before the merge webhook is processed, the default-branch graph stays as it was.
- A check run or apply comment GitHub refuses to edit is created again, so after the App is replaced the old App's checks stay on the commit beside the new ones, and a deleted apply comment reappears while its run is applying.
- The CLI retries
429and honoursRetry-Afterup to one minute; a429or5xxthat persists through the retries makesresolve,plananddriftunconfirmed andapplyexit 3, where a429exited 1. - The CLI derives the OIDC audience from the server URL without a trailing slash; an explicit
STACKORDER_OIDC_AUDIENCEis still sent as set. - Size the database's
max_connectionsforpool_max_conns+ 1 per server instance: the documentation saidpool_max_conns, though the server's use of connections is unchanged.
New stackorder-server subcommands, which need only DATABASE_URL: migrate (with --to N or --status), migrate --down --to N (a dry run without --yes), migrate --force N, and api-key create, api-key list and api-key revoke (with --actor, and --read-only, --installation and --expires on create).
New settings, each keeping the v0.5.0 behaviour by default: STACKORDER_SCHEDULER_CATCH_UP (1h), STACKORDER_SESSION_TTL (168h), STACKORDER_HSTS_MAX_AGE (unset, no header), STACKORDER_HSTS_INCLUDE_SUBDOMAINS and STACKORDER_HSTS_PRELOAD (false), STACKORDER_STATE_EXTERNAL_ID (unset) and GITHUB_ENTERPRISE (unset, inferred from GITHUB_WEB_URL). The Terraform module adds create_cluster (true), cluster_arn, database_secret_arn, app_secret_arn, metrics_token_secret_arn and shared_outputs_external_id (null) and capacity_provider_strategy ([]); an existing deployment plans no resource changes, as moved blocks carry the cluster and the three secrets.
New metrics: stackorder_runs_created_total{trigger,mode}, stackorder_artifact_operations_total{operation,result} and stackorder_pending_object_deletions. New endpoints: POST /v1/keys, GET /v1/keys and DELETE /v1/keys/{id} for administrator API keys, and PUT /v1/repos/{owner}/{repo}/default-graph for runner tokens.
Added
- An authenticated way to create, list and revoke API keys, so operators no longer insert rows into
api_keysby hand. The newstackorder-server api-key create|list|revokesubcommand manages keys from a shell withDATABASE_URLset and needs no App credentials, so an operator mints the first key when none exists;createprints the secret once, and--actor(defaultoperator) records the creator and the audit actor. Over HTTP,POST /v1/keys(201with the secret shown once),GET /v1/keysandDELETE /v1/keys/{id}do the same for a caller that already holds a key. A key's name is trimmed and may be at most 100 characters, counted as Unicode code points and enforced the same way by the subcommand and the endpoint. Managing keys is administrator-only: a signed-in browser user is403 forbiddenand a runner token401, so only an API key manages keys. Creating and revoking a key, by either route, is written to the audit log asapi_key_createdandapi_key_revoked, each committed in the same transaction as the change it records, so a key never exists without its creation entry and a revocation is audited only when it takes effect; revoking an already-revoked key keeps the first revocation time and records no further entry. The secret is a SHA-256 hash at rest and never reaches the logs. It needs no setting, and rows inserted intoapi_keysby hand keep working. See API keys. - Scoped API keys, so a key handed to a dashboard, a script or a support engineer need no longer be a standing administrator. A key may now be created read-only (refused every mutating route, enforced centrally by request method so even a route added later is covered), installation-scoped (sees only one installation — repositories, stacks, overview counts, runs, plan text, shared outputs, resources and the audit log — and treats everything of any other installation as
404 not_found, with no403-versus-404existence oracle), and expiring (refused exactly like a revoked key once past its time, checked on every request). The scope flags are accepted by bothstackorder-server api-key create(--read-only,--installation <id>,--expires <duration>) andPOST /v1/keys(read_only,installation_id,expires_at), and are shown byapi-key list,GET /v1/keysand in theapi_key_createdaudit entry. A scoped key is not an administrator: it can neither list, create nor revoke keys, so it can never obtain a broader key; only an unscoped read-write, installation-wide, non-expiring key manages keys, so a time-boxed key cannot mint a permanent one, andGET /v1/mereportsadminaccordingly. Migration0016_api_key_scopesadds the nullableinstallation_idandexpires_atcolumns andread_only(defaultfalse) toapi_keys; it is safe on a populated database and needs no superuser. Upgrading from v0.5.0 changes nothing for existing credentials: every existing key, and every row inserted by hand, stays an unscoped administrator that never expires, so no credential is narrowed or invalidated by the upgrade (only aNULLinstallation_idis unscoped — a hand-inserted0is a real, matchable installation scope). A scope is enforced only by a replica on this release, which stores a scoped or expiring key under the same hash v0.5.0 accepts while reading neither the scope columns nor the expiry, so during a rolling upgrade a still-running v0.5.0 replica honours such a key as a full, never-expiring administrator: mint and hand out scoped or expiring keys only once every replica runs this release, and revoke them before rolling back (see Upgrades and backups). See API key scopes, Scope a key and the security model. stackorder-server migrateapplies the embedded migrations toDATABASE_URLand exits, for operators who migrate as a separate deploy step or from a one-off job instead of relying on the automatic migration at server start-up. It takes the same migration lock as start-up, so it is safe to run beside starting instances: they apply each migration once. It needs onlyDATABASE_URL, notSTACKORDER_BASE_URLor the GitHub App settings.stackorder-server migrateapplies every pending migration (exit0, printing the versions migrated, ordatabase already at version N);stackorder-server migrate --to Nstops at versionNand refuses with exit1to move below the applied version or to a version that does not exist;stackorder-server migrate --statuschanges nothing and printscurrent=,target=anddirty=lines, exiting0when the database is up to date,2when migrations are pending and1when the database is dirty or ahead of the latest version the binary embeds, so--statusand plainmigrateagree on a database the binary cannot migrate up. Server start still migrates automatically, so an upgrade from v0.5.0 needs no new step; runningmigratefirst is optional. Run it from the container image withdocker run --rm -e DATABASE_URL=… ghcr.io/stackorder/stackorder:X.Y.Z migrate. See Migrating and Upgrades and backups.- Recovery from a migration that failed and left the schema
dirty. The server andstackorder-server migratenow refuse a dirty database with an error that names the dirty version, says a migration did not finish, and points at the recovery, instead of the bareDirty database version N. Fix and force version.from the migration library. The newstackorder-server migrate --force Nrecords versionN, clears the dirty flag and exits0, without running any migration; it only recovers a dirty database and refuses one that is not dirty (exit1), reading the dirty flag and recording the version in one locked operation so that a delayed, repeated or concurrent--forcecannot rewind an already-recovered database, and it cannot skip migrations on a healthy schema. Each migration file runs in one transaction under thepgx/v5driver, so a migration that fails rolls back and leaves the schema at the version before it: after migrationNfails use--force N-1and runmigrateagain to re-applyN, or--force 0for a failed first migration; forceNitself only when you completedNby hand. This is a new flag and a clearer message only — there is no new migration or setting, and a database that has never had a failed migration is unaffected, so upgrading from v0.5.0 needs no action. See Recover from a dirty migration and Migrating. stackorder-server migrate --down --to Nreverts the schema down to versionN, so an operator can roll back to an older release without restoring a snapshot when the migrations in between were additive. Until now the server only migrated up and an older image refused to start on a newer schema, so going back meant restoring the pre-upgrade snapshot and losing every write since. Run it with the newer image, which embeds the down migrations the older one lacks; it is guarded — without--yesit prints the plan, the migrations it would revert and the data each one drops, and changes nothing, and only--yesreverts. It takes the same migration lock as start-up and warns to stop every instance of the newer release first, since a running one fails once the schema it uses is gone and the older release refuses to start until the schema is at or below the version it embeds. It refuses with exit1a target above the applied version,--to 0, adirtydatabase, and a database ahead of the binary (whose migrations it cannot revert). Every embedded migration now has a tested down file, checked by a test that fails when one is missing and by up/down/up round trips on a populated database. This adds no setting and changes no default: automatic migration at start-up is unchanged and only ever migrates up. See Migrating and Rolling back, which tabulates each release's schema version and what a downgrade loses.STACKORDER_SCHEDULER_CATCH_UPsets how far a new scheduler leader looks back to catch up schedules missed while no instance was leading, a positive Go duration defaulting to1hand capped at168h. The scheduler already enqueues only the latest fire of each schedule in the window, so a window wider than an outage recovers the dailystale_locks(08:00UTC) andsync_installations(04:00UTC) jobs and each stack's latest drift fire without a burst of repeated work; a server down for longer than this window still drops those jobs until their next scheduled time. The default preserves the previous fixed one-hour look-back, so upgrading from v0.5.0 changes nothing unless the window is set; set it, for instance, to26hto recover a full day's maintenance and drift schedules on restart. Reachable through the Terraform module'sextra_environment. See Server configuration.STACKORDER_SESSION_TTLsets how long a browser session lasts, a Go duration between5mand8760h, defaulting to168h(7 days), the previous fixed lifetime. It fixes both the session cookie'sMax-Ageand the server-sideexpires_atat sign-in, so a change applies only to sessions created after it; existing sessions keep the lifetime they were issued with. See Server configuration.STACKORDER_HSTS_MAX_AGEadds aStrict-Transport-Securityheader to every response, with its value as themax-age, a positive Go duration of at least1ssuch as8760h.STACKORDER_HSTS_INCLUDE_SUBDOMAINSandSTACKORDER_HSTS_PRELOAD(bothfalseby default) addincludeSubDomainsandpreload;preloadrequiresincludeSubDomainsand amax-ageof at least one year. The header is sent only whenSTACKORDER_BASE_URLishttps, the same signal that marks the session cookieSecure; settingSTACKORDER_HSTS_MAX_AGEwith anhttpbase URL, or the sub-settings without it, is a start-up error. It is off by default, so a self-hoster on plain HTTP or on a shared parent domain is not locked into HTTPS by an upgrade. See HSTS and Security hardening.STACKORDER_STATE_EXTERNAL_ID(unset by default) sets thests:ExternalIdthe server sends when it assumesSTACKORDER_STATE_ROLE_ARNto read shared outputs state, so an operator whose state role lives in another account can require a matching external ID in the role's trust policy to guard against the confused deputy. It must be 2 to 1224 of the characters AWS allows in an STS external ID (letters, digits and+=,.@:/_-), and setting it withoutSTACKORDER_STATE_ROLE_ARNstops the server at start. The external ID is an identifier, not a secret, and the server never logs its value; without it the role assumption is unchanged, so an existing deployment upgrading from v0.5.0 needs to do nothing. The Terraform module reaches it through the newshared_outputs_external_idinput (nullby default), which requiresshared_outputs_role_arn. See Security hardening for a trust-policy example.GITHUB_ENTERPRISE(trueorfalse, unset by default) forces whether the server is treated as talking to GitHub Enterprise Server, which decides the App manifest/setupoffers. Until now that was inferred only fromGITHUB_WEB_URL, so a server reaching github.com through a proxy or mirror whose web URL is nothttps://github.comwas offered the Enterprise Server manifest, without thedeploymentspermission and thedeployment_protection_ruleevent. SetGITHUB_ENTERPRISE=falseto keep the full github.com manifest in that case, orGITHUB_ENTERPRISE=trueto request the Enterprise Server manifest even onhttps://github.com. Unset, the behaviour is unchanged: anyGITHUB_WEB_URLother thanhttps://github.com(case and a trailing slash ignored) is treated as Enterprise Server. See Server configuration.- Terraform module: optional inputs to run on infrastructure you already have, each defaulting to today's behaviour so an existing deployment plans with no changes.
create_cluster(defaulttrue) withcluster_arnruns the service in an existing ECS cluster instead of creating one.capacity_provider_strategy(default[]) runs the tasks under a capacity provider strategy, FARGATE and FARGATE_SPOT with weights and an optional base, in place oflaunch_type = "FARGATE", which ECS forbids alongside it; at least one provider must have a positive weight, since ECS places no tasks when every weight is zero and an omitted weight is zero; for a cluster the module creates it associates the strategy's providers, an existing cluster must already have them.database_secret_arn,app_secret_arnandmetrics_token_secret_arn(defaultnull) make the module write each secret value, still with a write-only argument so nothing secret reaches state, into a Secrets Manager secret you own instead of creating<name>/database-url,<name>/appand<name>/metrics-token. An existing VPC was already supported throughcreate_vpc,vpc_id,public_subnet_idsandprivate_subnet_ids; an existing database or load balancer stays out of scope. Upgrading from v0.5.0 moves the ECS cluster and the three secrets undercountthroughmovedblocks, so no resource is recreated and no state surgery is needed. See Deploy on AWS. - A direct push to the default branch can now record its graph as the repository's default-branch graph, so the stack lists,
GET /v1/overview, removed stacks and drift scheduling follow the default branch at once instead of waiting for the next merged pull request that had a plan. The new runner endpointPUT /v1/repos/{owner}/{repo}/default-graphtakes the graph a push-triggeredresolvejob scanned and makes it the default-branch graph, reconcilingstacks.removed_atexactly as a merged pull request's plan does. The server accepts it only from a GitHub Actionspushtoken whoserepository,repository_idandref(refs/heads/<default branch>) are the repository's and whoseshais the graph's, and only while that commit is still the branch head, which it reads from GitHub under a per-repository lock; an out-of-order, superseded or force push, whose commit is no longer the head, records nothing (recorded: false), so an older push can never replace a newer graph. Thestackorder resolvecommand records the graph instead of resolving a pull request when it runs on a push to the default branch. No server setting changes and no migration is added; nothing records a graph on a push until a repository triggers its plan workflow on a push to the default branch, which nostackorder/actionsrelease does yet (v1.2.0 does not), so an existing deployment keeps the previous behaviour, where only a planned merge moves the default-branch graph. See Workflows and the CLI reference. - The server carries trace context across the work queue, so a webhook delivery and the worker that handles it, and a job and the worker that runs it, are one trace instead of two. When tracing is on (
OTEL_EXPORTER_OTLP_ENDPOINTset), the W3C trace context of the request or job that queues a row is stored in a new nullabletrace_contextjsonb column oneventsandjobs(migration 0015), and the worker span is started as a child of that remote parent, sharing its trace id; a retry is a new child span under the same trace, and a span that begins long after its parent ended shows the queue delay. Log records written inside a span, thehttp requestaccess log and the event and job handlers included, now also carrytrace_idandspan_id, so a log line joins its trace and, through the stored context, the request or job that queued the work. With tracing off nothing changes, and no log line gains the fields. Upgrading from v0.5.0 applies migration 0015, which adds a nullable column on populatedeventsandjobstables without rewriting them and needs no superuser; rows queued before the upgrade have no context and their worker spans stay root spans, and a server of this release and one of v0.5.0 keep running side by side during a rolling deploy, each inserting and claiming rows. Once 0015 is applied, as with every migration of this release, a v0.5.0 server can no longer start against the database: its startup migration ends at 0009 and refuses a schema it does not recognise, so a rollback to the previous image, or a restart of a drained v0.5.0 replica, needs the schema reverted first with this release'sstackorder-server migrate --down --to 9 --yes, or the pre-upgrade snapshot restored. See Upgrades and backups, Metrics and tracing and the data model. - The
stackorder_artifact_operations_total{operation,result}counter records reads and writes against the optional artifact bucket, so a broken bucket policy or missing credentials no longer degrades silently to truncated plan text in Postgres.operationiswriteorreadandresultisokorerror; a bucket write that fails countsoperation="write",result="error"(a failed plan-text write is what forces the fallback to Postgres), and a read of an object that is gone counts asok, as the store answered. The counter appears only withSTACKORDER_ARTIFACT_BUCKETset, and there is a sampleStackorderArtifactWriteErrorsalert. See Metrics. stackorder_runs_created_total, a counter of runs created, labelledtriggerandmode, incremented once per run row at every place a run starts, bytrigger:pull_requestfor a pull request's plan and for anon_mergeapply after the merge,commentfor aplanorapplycomment,rerequestfor a re-run requested from a check run on GitHub or withPOST /v1/runs/{id}/rerun,pushfor a cross-repository plan of the stacks that depend on an upstream apply,manualfor a manual apply andschedulefor a scheduled drift check, and never for a run that was reused or whose insert rolled back. Use it to count runs; summingstackorder_runs_total, which counts each status a run enters, overcounts them. See Metrics.stackorder validatechecks the rootstackorder.yamland every.stackorder.yamlfile and the dependency graph without a server or credentials, for a pull request or a pre-commit hook before anything runs. It reports everythinggraphandaffectedreject — unsupported versions, unknown keys, invalid globs and patterns, and dependency cycles — and every warning a scan reports, each as one diagnostic with a severity, a message and, where the loader knows them, the file and the line. It parses every configuration file before it stops, so one run reports the problems of several files at once, and skips the graph checks while a file does not parse. The text output ends with aninvalid: N errors, M warningsorvalid: …line, orfailing: … (--strict)when--strictfails a run that has only warnings, so the last line always agrees with the exit code;--format jsonprints a stableValidationReportofvalid, theerrorsandwarningscounts and thediagnostics; in GitHub Actions it also writes each diagnostic as a::error file=…,line=…::or::warning::workflow annotation. It exits 0 when there are no errors and 1 when there is at least one;--strictalso exits 1 when there are only warnings. See the CLI reference.
Changed
- A failing
.stackorder/hooks/post-apply.shnow fails its stack instead of reporting it as applied. The apply itself still ran and its resources and state are still reported, but the CLI posts the result asfailurewith the hook's exit code and output, so the server records the stack asfailed, blocks its dependents in later waves, does not dispatch them, keeps the run's orchestration locks held for abefore_mergeapply, and marks the run failed; the command still exits 1. Before this release a post-apply hook, typically a smoke test, failed the job's exit code but let the run continue and apply every dependent stack. This is a behaviour change with no setting: a post-apply hook that fails today with no consequence now blocks the stacks that depend on it. Server and CLI can still be upgraded in either order, since the resultstatusvalues are unchanged: an older server already mapsfailureto a failed stack, and an older CLI keeps the previous behaviour. See Hooks. - The
stackorder_github_rate_limit_remaininggauge now carries aninstallationlabel and is absent until the first GitHub response, where before it was a single unlabelled series registered at0. GitHub's rate limits are per installation, and the App JWT has its own, so one unlabelled gauge mixed every budget and let one installation's remaining count overwrite another's; a value is now recorded per installation id, and underinstallation="app"for the App JWT. Requests made with a user access token (the sign-in flow) are not tracked. Because the series only appears once a response has been seen, it is no longer0on a freshly started, idle server, so the sampleStackorderGitHubRateLimitLowalert no longer fires at start-up; that alert's expression becomesmin by (installation) (stackorder_github_rate_limit_remaining{installation!=""}) < 500. Upgrading from v0.5.0: a dashboard or alert that reads the unlabelledstackorder_github_rate_limit_remainingmust select a label, for examplemax by (installation) (...)or...{installation="app"}; the sample alert matchesinstallation!=""so a v0.5.0 replica still exporting the unlabelled series at0during the rolling upgrade does not trip it. A series for a removed installation lingers until the server restarts. See the metrics reference. - A merged pull request now records its plan graph as the repository's default-branch graph only while the merge commit is still the head of the default branch, read from GitHub under the same per-repository lock a direct-push recording takes, so the two writers share one ordering. Before this release the merge path set
repos.default_graph_idunconditionally, so a delayed or redeliveredclosed/merged webhook — one for an earlier pull request processed after a later merge, or after a direct-push recording — could replace the current default-branch graph with an older one, reverting the stack lists,GET /v1/overview, removed stacks and drift scheduling until the next graph change. A webhook whose merge commit is no longer the branch head now records nothing. Releasing that pull request's orchestration locks (before_merge, the default) no longer waits on the head read: the locks are released before the graph is recorded, so a transient GitHub ref-read failure, where the merge path now reads the branch head, no longer keeps the pull request's stacks locked; the queued merge event fails and the server's queue retries it, after 1 minute, then 5 and 30 minutes and 2 hours, and the graph is recorded on the retry. No setting changes and no migration is added; a normal merge, whose merge commit is the branch head when the webhook arrives, still records its graph. Upgrading from v0.5.0: where a repository's own automation pushes a further commit to the default branch before theclosed/merged webhook is processed (a version bump, a changelog commit or a merge queue landing a follow-up), the merge commit is no longer the head when the webhook arrives, so the merge now records nothing where v0.5.0 advanced the default-branch graph to the pull request's graph unconditionally; the graph then stays as it was until the next recording — the follow-up push's own, once the repository triggers its plan workflow on a push to the default branch, which nostackorder/actionsrelease does yet. SeePUT /v1/repos/{owner}/{repo}/default-graph. - The CLI retries
429 Too Many Requestsfrom a rate limiter or reverse proxy in front of the server, which it failed on at once before, alongside the network errors, timeouts and 5xx answers it already retried, and honours aRetry-Afterheader in either its seconds or its HTTP-date form on any retried answer, capped at one minute (a header asking for longer stops the retries). A429or 5xx that persists through the three retries is now unreachable, soresolve,plananddriftdegrade to anunconfirmedresult andapplyfails closed (exit 3) where a429failed with exit 1 before. Upgrading needs no configuration: the attempt count, the 200 ms/800 ms/2 s backoff when noRetry-Afteris sent, and the 30 s per-attempt timeout are unchanged, and each attempt still requests a fresh OIDC token so a retry never replays one. See the CLI reference and Exit codes. - The documentation of
stackorder_runs_totalnow says it counts each status a run enters, a transition count rather than a run count, and points at the newstackorder_runs_created_totalfor counting runs. The metric itself is unchanged. - The documentation site at docs.stackorder.io loads Cloudflare Web Analytics, a cookieless visit counter, on every page. Nothing in the CLI, the server image or the Terraform module changes.
Fixed
- The server no longer retries a GitHub call that creates something new — a check run, a comment, a drift issue, or a
workflow_dispatch— once the request may already have reached GitHub, so a lost response or a5xxafter the request was sent surfaces as an error instead of silently creating a second check run, comment, issue or workflow run (a duplicate the self-hoster would see as two check runs of one name, a reminder posted twice, two drift issues, or a second workflow run that uses runner time before its jobs are refused). Such a call is still retried while the request provably never left (a connection error before it was written) and when GitHub reports a rate limit and did no work (429, or a primary or secondary rate limit). Reads,PUT/DELETE, thePATCHof a check run, comment or issue, the reactionPOST, the deployment-protection-rule reviewPOSTand the installation-token exchange stay idempotent and are retried as before. When a check-run create's outcome is unknown, the renderer now adopts the check run carrying the run'sexternal_idinstead of creating a second one; the run comment and sticky comment are found again by their marker, a drift issue by its title and a fork notice by its check-run name before a create; a stale-lock reminder records its audit row and a resolved drift issue is closed before the comment is posted; and an unsent dispatch is resent once by Reconcile, where run binding refuses the surplus workflow run as before. The duplicate check run, comment, issue and fork notice are now prevented outright, while a resent dispatch's surplus run, which can still start, is made harmless by binding rather than silently multiplied by the transport. Nothing an operator sets changes; the retry count and backoff are unchanged. See the GitHub App reference. - The server now recreates a check run or an apply run's comment when GitHub refuses to edit the one it recorded because the object is gone or no longer the App's to edit — a
403that is not a rate limit, a404or a410— where it previously fell back to creating a new check run only on404and, for a comment, recordedcomment_id = 0and never posted again. After the App of a deployment is replaced (a new App registration, or a database restored onto a server configured with a different App), GitHub answers edits of the old App's objects with403, so the checks and comment of every open run were stuck; they now recover on the run's next update, each getting new check runs and a new comment under the new App. The new comment is found by its marker, so the new App posts at most one and reuses it on later updates; each new check run's id is recorded the same way. The old objects, which the new App can neither edit nor delete, are left on the commit, so a check run can appear twice there after a replacement. A rate limit or a5xxstays transient and never creates a duplicate. A recordedcomment_idof0no longer suppresses a live run's comment: it now marks only a finished run that has no comment and gets none, and a running apply whose comment was deleted is posted again, as the sticky comment already was. No migration is needed — each run re-derives its check runs and comment on its next render — and no setting changes; the one difference an upgrader from v0.5.0 will see is that a deleted apply comment reappears while its run is still applying. - A stack deleted from a repository's default branch is now marked removed, so it leaves
GET /v1/stacks,GET /v1/repos/{owner}/{repo}/stacksand the Stacks page, and stops being counted byGET /v1/overviewand in each repository'sstacksanddrifted. Whenever the default-branch graph changes, the server reconcilesstacks.removed_atto the change: a stack the previous default graph listed and the new one does not is stamped removed, and a stack the new one lists hasremoved_atcleared, while a stack never on the default branch, such as one an open pull request proposes, is left listed. Only a change of the default-branch graph clearsremoved_at: saving a plan for an open pull request whose branch still has the deleted directory no longer brings the stack back, so the removal is durable until the directory actually returns to the default branch, and concurrent merges to one repository reconcile in order. This durability holds once every server replica runs this release: during a rolling upgrade from v0.5.0 a still-running v0.5.0 replica keeps clearingremoved_aton every plan save, so an open pull request replanned by an old replica can revive a stack already deleted from the default branch, and because that stack is no longer on the default branch no later merge re-marks it — it stays listed until its directory returns to the default branch, so complete the rollout promptly (see Upgrades and backups). Before this releaseremoved_atwas never set, so the readers that filter on it never hid anything and a deleted stack stayed listed and counted forever. Migration0010_reconcile_removed_stacksrepairs existing databases on upgrade from v0.5.0, marking removed every stack absent from its repository's current default-branch graph unless a newer graph still lists it; it touches no stack of a repository that has not recorded a default-branch graph yet, needs no superuser and is safe on a populated database. A removed stack keeps its history, runs and last resource list, and is still served by id atGET /v1/stacks/{id}; it reappears if a later merge brings its directory back. Unless the repository records its default-branch graph on direct pushes (see the entry under Added), the graph moves only when a planned pull request merges, so a stack deleted by a direct push is marked removed at the next such merge. - With an artifact bucket, the server now deletes a stack's
plan.txtandplan.jsonobjects whenSTACKORDER_PLAN_TEXT_RETENTIONprunes its plan text, and deletes every plan object of a repository when the repository is removed by webhook, installation removal or sync, so the bucket no longer grows without bound and the full plan text stops being served once its retention clears the beginning kept in Postgres rather than only when a bucket lifecycle rule expires the object. Clearing a stack's plan text also clears itsplan_urlin the same statement and queues the objects in the newobject_deletionstable (migration 0012), which the prune job sweeps; queuing outlives the run, so a crash between the database change and the deletion loses neither. Pruning also reclaims a stack whose plan text an older server cleared while leaving itsplan_url, so plan text a v0.5.0 server already pruned, and plan text a replica still running the previous release prunes during a rolling upgrade, are queued and deleted on the next prune rather than being stranded in the bucket until a lifecycle rule expires them (migration 0013 indexes those rows). Removing a repository or an installation queues the objects from the stack rows it deletes, so aplan_urla result records up to the moment of deletion is captured rather than escaping through the cascade. Each sweep attempts a given object once, so a repeatedly failing delete no longer inflates its attempt count or the warning-log volume within one sweep. Deletion is always on and needss3:DeleteObject; a role without it keeps working, because a failed delete never fails pruning or repository removal: the object is logged, left in the bucket, retried on the next prune and counted in the newstackorder_pending_object_deletionsgauge. Upgrading from v0.5.0: a deployment whose IAM policy followed earlier documentation and granted onlys3:GetObject,s3:PutObjectands3:ListBucketshould adds3:DeleteObjectto reclaim space, including the objects of plan text v0.5.0 already pruned; the Terraform module already grants it. Keep a bucket lifecycle expiry as a backstop for objects a role could not delete. See the data model and metrics. - An installation's suspension no longer depends on the order the workers happen to process its webhooks. Several workers and replicas claim queued events with no ordering per installation, so a
suspendand anunsuspendof one installation could take effect out of receive order, or a failedunsuspendbe retried after a latersuspend, leaving the installation suspended — its webhooks ignored, its runner tokens refused and no plans started — until the daily sync corrected it. The server now records, in the newinstallations.suspend_observed_atcolumn (migration0014_installation_event_order), the observation time of the lastsuspend/unsuspendapplied, and applies one only when its time is no earlier than the last, as a single compare-and-set, so the suspension converges on the most recently observed delivery whatever order the workers ran. Asuspendprocessed before the installation's owncreatedis now recorded against the installation rather than dropped, so a latercreatedwith an earlier receive time no longer leaves it active; a consequence is that asuspendretried after the installation'sdeletedevent re-creates the installation as an inert, suspended row (thecreatedandunsuspendpaths already recorded the installation unconditionally in v0.5.0), which the daily sync removes within a day along with any installation GitHub no longer lists. Acreated,unsuspendornew_permissions_acceptedthat loses the compare-and-set still records the repositories it carries unless the installation is now suspended, in which case its token cannot read them. The daily installation sync records each suspension with the database time it read before listing, closing a narrower version of the same race where anunsuspendreceived during a sync was undone until the next day. This is a behaviour fix with no new setting. The migration adds a nullable column and is safe on a populated database upgraded from v0.5.0; events already queued are applied in receive order without a backfill. The receive-order guarantee holds once every server replica runs this version: during a rolling upgrade a v0.5.0 replica still applies the lastsuspend/unsuspendit processes, as it did before, and the per-day sync and the finished upgrade reconcile it — see Upgrades. See the data model. - The installation sync no longer deletes a stored installation, with its repositories and their whole history, just because one
GET /app/installationslisting left it out. A proxy in front of the GitHub API, a GitHub incident or a pagination fault could return an incomplete or empty200and cascade-delete everything the server knew. The sync now forgets an omitted installation only afterGET /app/installations/{id}confirms it with a404; a200keeps it, and a5xx,403, timeout or any other answer keeps it and retries at the next sync. A listing that omits a majority of the stored installations is treated as degraded and keeps every one of them, logging a warning; their repositories are still reconciled from each listed installation's own repository listing. A listing that omits a majority of a listed installation's repositories keeps that installation's repositories, and because a repository has no per-repository confirmation, a listing that omits every one of them keeps them as well, so an empty or short200cannot delete a single-repository installation's only repository; omission is measured against the installations and repositories stored before the sync, so repositories it newly discovers do not dilute it. A repository listing whose pages hold fewer repositories than thetotal_countGitHub reports counts as a failed listing, and because any failed listing makes the whole sync forget nothing, a dropped page keeps every stored installation and repository, not only the installation whose pages were short. Theinstallationdeletedandinstallation_repositoriesremovedwebhooks still forget installations and repositories at once. This changes only when the periodic sync deletes: no new settings, no migration. See the GitHub App reference. - A data-only dump (
pg_dump --data-only) of a populated database now restores into a freshly migrated one in a single transaction without a superuser.reposandgraphsreference each other, and no row-by-row load order satisfied both foreign keys, sopg_restore --data-onlyfailed unless triggers were disabled, which needs a superuser managed Postgres does not grant. Migration 0011 makesrepos.default_graph_id's foreign keyDEFERRABLE INITIALLY DEFERRED, so the cycle is checked once at commit; load the whole dump withpsql --single-transaction(orpg_restore --single-transaction). TheALTER CONSTRAINTchanges the catalog only and does not revalidate the table. Server behaviour is unchanged: the constraint is still enforced, and the delete-timeSET NULLstill fires immediately. See Upgrades and backups. - The CLI now removes a trailing slash from the OIDC token audience it derives from the server URL, matching the audience the server derives from
STACKORDER_BASE_URL. A server URL written with a trailing slash (--server/STACKORDER_SERVER_URL, for examplehttps://stackorder.example.com/) made every runner call fail with401, because the runner token'saudkept the slash while the server's expected audience had it removed. An explicitSTACKORDER_OIDC_AUDIENCEis still sent as set, with only surrounding whitespace trimmed and any trailing slash kept, matching how the server reads it. No configuration changes; upgraded runners that previously needed a slash-free server URL now work either way. - A failing hook's own output now reaches the result
error_text, so apre-plan,post-plan,pre-applyorpost-applyhook's final diagnostic, such as a smoke test'ssmoke test: endpoint returned HTTP 503, appears in the job step summary and not only the streamed job log. Before this release the error text carried only the hook's exit code. The CLI keeps the tail of the hook's output, redacts it as it redacts tool output, and cuts it to fit the result body. - The
/setupdone page no longer tells everyone to require both thestackorder/planandstackorder/applychecks in branch protection. Requiringstackorder/applyis right only whereapply.modeisbefore_merge, the default; inon_mergemode the apply runs after the merge, so a requiredstackorder/applycheck would block every merge. The page now states that condition, matching the GitHub App reference. The same correction is made in the README and the design notes, which gave the advice unconditionally. - The documentation and a code comment said the scheduler's leader lock holds one of the connection pool's connections. It does not: each server instance opens one dedicated connection outside the pool for the leader lock (the leader holds it; a non-leader opens and closes one every 30 s to test the lock), so a running instance uses at most
pool_max_conns+ 1 connections. The start-up migration lock takes a dedicated connection the same way, before the scheduler runs, so the two never overlap. Running the server, Server configuration andARCHITECTURE.mdnow state this per-instance budget; when upgrading from v0.5.0, size the database'smax_connections, or any connection pooler in front of it, forpool_max_conns+ 1 per server instance rather thanpool_max_conns.
0.5.0 - 2026-11-02
Added
- A stack's page lists the resources in its state, grouped by module and filtered by address, with the applied run and pull request that last created, updated or replaced each one, and marks those the latest drift check found drifted and those that are tainted or have a deposed object. Apply jobs once the tool has run, failed applies included, and drift jobs after their plan read the state with
show -jsonon the runner and report it asresourcesin their results: the address, type, module and provider of each managed resource and its tainted and deposed flags, never an attribute value, output or identity, and no data source. The CLI captures the state without streaming it to the job log or writing it to disk, redacts the list as it redacts plan text, cuts it to fit the 1 MB result body, keeping the first resources by address, and only warns when the read fails; when the server answers413, or400 invalidwith adetails.fieldunderresources, it posts the result once more without the list. The list is part of--format jsonoutput, never ofGITHUB_OUTPUTor the step summary. Plan jobs never send one, andplan_outputdoes not change it. See the CLI reference. - The server keeps each stack's latest resource list in the new
stack_resourcestable (migration 0009, which also indexes the applied rows ofrun_stacks) and serves it atGET /v1/stacks/{id}/resourcesto people who see the repository and to API keys, never to runner tokens. Each item'slast_changecomes from the plan summaries of the stack's applied runs, which the server already keeps, so imports, moves and failed applies are never attributed. A list from an accepted apply or drift result, manual runs included and whatever the outcome, replaces the recorded one unless that one was read from newer state: a confirmed apply result counts as read when it first arrives and, when it arrives again, when its stack finished, a drift or unconfirmed apply result as read when its run was created, so a drift check created while an apply waited for approval never replaces the apply's list. The server drops invalid items and marks the list truncated rather than refuse a result, and a list it cannot record never fails the result. The list is reported by jobs that run the repository's code, a pull request's for an apply before merge, so it is not proof of what exists; see the security model. The server and the CLI can be upgraded in either order: an older server ignoresresources, and with an older CLI a stack's page shows no resource list until a job on this release reports one, which a scheduled drift check does without changing anything. GET /v1/stackslists the stacks of every repository the caller can see, ordered by repository and then key, as aPageofStackDetailwhose items are thoseGET /v1/repos/{owner}/{repo}/stacksreturns. It pages withlimitandcursor, carriestotal, and leaves out stacks marked removed.- The web UI has a Stacks page at
/stacks, reached from a Stacks item after Repositories in the main navigation and from the stack count on the overview. It lists every stack with its repository, environment, last apply, drift and lock, and?q=filters it by key, repository, path or environment. A stack page now marks Stacks, not Repositories, as the current section.
Changed
- A JSON value of the wrong type in a request body answers
400 invalidwithdetails.field, the field's path without array indexes such asresources.items.tainted, anddetails.reason, where only the message named the field before. See Errors. - The documentation no longer says that the server stores only where each stack's state lives, nor that
plan_output: summarysends only counts and addresses: the server also stores each stack's resource list, still never the state or an attribute value from it, andsummarysends no plan text, while apply and drift jobs still send the list.SECURITY.mdcounts a state value that reaches the server through the list as in scope. - Reading outputs in Terraform links the
stackorder/stackorderprovider on the Terraform Registry and the OpenTofu Registry, where v0.1.0 is published, pins it withversion = "~> 0.1.0", and points atstackorder/example-infra, whosestacks/prod/eksreads the outputs ofstacks/prod/vpcwith it. The local demo and the CLI reference show that stack's inferredreads_stateedge in place of itsdepends_on, and the README andARCHITECTURE.mdlist the provider's repository.
Removed
- The live variant of the end-to-end suite:
TestLiveGitHub, the packagetest/e2e/live, thelivejob ofe2e.ymland theSTACKORDER_E2E_GITHUB_TOKEN,STACKORDER_E2E_ORG,STACKORDER_E2E_SERVER_URL,STACKORDER_E2E_PLAN_ROLE_ARN,STACKORDER_E2E_GITHUB_API_URL,STACKORDER_E2E_GITHUB_URLandSTACKORDER_E2E_KEEP_REPOsettings. It ranstackorder/example-infraagainst a real GitHub organisation and server. The example now runs only against LocalStack, in the end-to-end suite, with the in-process server and the fake GitHub, and in the local demo.
Fixed
SECURITY.mdno longer says only the 0.2.x releases receive security fixes; the latest release does. The security model and the App's permissions say thatContents: Readlets the server, and anyone who compromises it, read every file of an installed repository, not only the Stackorder configuration.
0.4.0 - 2026-10-11
Added
- Shared outputs: a stack can read another stack's root-module outputs through the server instead of
terraform_remote_state, so its role needs no access to the other stack's state object. The producer lists its readers underoutputs.readersinstackorder.yaml, in its.stackorder.yamlor on an instance, as doublestar patterns over stack keys, orowner/repo//patternfor stacks of another repository of the same installation. The most specific level that setsreadersreplaces the others,[]withdraws a broader grant, and unset at every level the outputs are not shared, inside one repository too. The server reads the producer'soutputsconfiguration from its default branch only, never from a pull request or a graph upload. Outputs marked sensitive are left out and named inwithheldunlessoutputs.sensitiveisshare; the default,withhold, keeps them out of the consumer's saved plan and state. GET /v1/repos/{owner}/{repo}/stacks/{key}/outputsserves a producer's outputs to the runner jobs of the stacks it lists, which name their run and their stack, and to API keys, which bypassreaders; a session is refused. The server reads the producer's state object from S3 with its own AWS credentials, or through the role inSTACKORDER_STATE_ROLE_ARN, and only when the object lies under an entry ofSTACKORDER_STATE_BUCKETSgranted to the producer's repository. That variable, which turns the feature on, takes comma-separatedowner/repo=bucketorowner/repo=bucket/prefixentries, or*=bucket[/prefix]to grant an entry to every repository; a prefix covers a key only at a path segment boundary, sostacks/proddoes not coverstacks/production/. An apply or drift result whose state object no entry of its repository covers is accepted without recording the object and clears the stack's recorded location, so reads answer409 state_location_unknowninstead of serving the object the stack left, and one repository cannot point its stacks at another repository's state unless the entries of the two overlap, which the server warns about at start-up; a read of such a location answers403 state_bucket_not_allowedbefore any request to S3, and is audited and logged. Unset, every read answers404 outputs_disabled. It decodes only theoutputsblock of a version 4 state and stops reading there, refuses state that OpenTofu encrypted, and stores, caches and logs nothing it reads. A reader in another repository gets404 not_foundfor every refusal of the producer or its policy, so the endpoint tells it nothing about which stacks exist, and an unlisted stack of the producer's repository gets403 outputs_not_shared. Every read, and every refusal after the server accepts the token or key, of the run, the producer, its policy or its state location, is recorded in the audit log asoutputs_readoroutputs_refused, with counts and never values. See Server configuration.- The
stackorderTerraform provider, from stackorder/terraform-provider-stackorder, reads shared outputs in place ofterraform_remote_state, so a consumer's AWS role needs no access to the producer's state. Itsstackorder_outputsdata source and ephemeral resource take the producer's key asstackand returnvaluesandnonsensitive_values, and the ephemeral resource keeps them out of the saved plan and the state. Inside a Stackorder job the provider needs no settings. Its publication to the Terraform and OpenTofu registries follows this release. See Reading outputs in Terraform. - The scanner infers a
reads_stateedge withsource: stackorder_outputsfrom eachdataorephemeralstackorder_outputsblock whosestackis a literal string, in a.tfor.tf.jsonfile of a stack's directory or of a local module the stack calls, at any depth, so ordering, propagation and cross-repository dependents treat it as they treatterraform_remote_state. A key of the same repository must name a stack exactly,owner/repo//keyadds the producer as an external stack, and astackbuilt from an expression infers nothing and warns to list the producer independs_on. In a module, astackthat refers only tovar.<name>values and calls no function takes them from the literal arguments of eachmoduleblock that calls the module, one level up, and a block whosestackstill names no stack warns, naming the module's directory, that onlydepends_onorders the stacks that call the module. Blocks in registry and git modules are not read, andterraform_remote_stateis read only in a stack's own directory. Anignore_inferredentry for another repository that no such block reads is a warning. See Ordering. - The CLI reports the S3 object each job initialised as
statein its results: the bucket, the full key with the workspace prefix, and the region, afterbackend_config,STACKORDER_BACKEND_CONFIGand the workspace the tool uses, for ans3backend with a bucket and a key. The server records it from apply and drift results only, in new columns ofstacks(migration 0008), as the location it reads the stack's outputs from; a plan result or a graph upload never moves it. Terraform, OpenTofu and the hooks also getSTACKORDER_SERVER_URL,STACKORDER_RUN_IDandSTACKORDER_REPOSITORY, each set even when empty, so a value inherited from the job never disagrees with what the CLI uses, and the provider can call the server from a stack's job. - Terraform module:
shared_outputs_state, a list of buckets, each granted to onerepository, or to every repository with*, with an optional key prefix and KMS key, setsSTACKORDER_STATE_BUCKETSasrepository=bucket/prefixentries and grants the task roles3:GetObjecton the objects under each prefix,s3:ListBucketlimited to the same prefixes, so that a missing object answers not found rather than access denied, andkms:Decryptthrough S3 only; nothing in it can write or delete.shared_outputs_role_arnsetsSTACKORDER_STATE_ROLE_ARNand grants the task role onlysts:AssumeRoleon that role, whose own policies then grant the reads, for instance in the account of the state buckets. IAM grants the prefixes for every repository at once; the server's check keeps each repository to its own entries. The module refuses a repository that is neitherowner/reponor*, and a prefix with a.or..segment. Both are off by default, andextra_environmentrejects the two variables.
Changed
- The local demo needs Docker, git and Terraform or OpenTofu, and no longer Go. The
serverprofile ofdocker-compose.ymlruns the published image,ghcr.io/stackorder/stackorderat the tag inSTACKORDER_VERSIONorlatest, and the guide installs the CLI from the release archive after checking it against the checksums file.--buildstill builds the image from the checkout, and From source keepsmake devandmake build-clifor contributors. - With shared outputs on, the server's AWS role, or the role in
STACKORDER_STATE_ROLE_ARN, can read the whole state objects under the buckets and prefixesSTACKORDER_STATE_BUCKETSlists, secrets included: IAM grants whole objects, and the filtering to outputs happens in the server. Each entry is granted to one repository, so a repository's jobs can make the server read only state under its own entries; within one repository, an apply, including a pull request's apply before merge, can point one stack at another stack's state under the same entries. A compromised server could then read those objects and the secrets in them, and answer reads with values of its choosing, though it still cannot write state. The security model states the trade and the controls that bound it, Security hardening has the narrow read-only role, and the README,SECURITY.mdand the documentation qualify their claims that the server holds no cloud credentials and never reads state. A stack that reads shared outputs also calls the server from its plan, so that plan fails while the server is unreachable. Shared outputs are off by default, and the server logs once at start-up when they are on. - Upgrade the CLI before adding
outputsto astackorder.yamlor.stackorder.yaml: older CLIs reject unknown keys, which fails the resolve job of every pull request. The reusable workflows install the latest release unlessstackorder-versionpins one. A producer's outputs become readable once an apply or drift job on the new CLI has reported its state object; until then a read answers409 state_location_unknown, and a scheduled drift check records the object without changing anything.
Fixed
make devwaits until Postgres and LocalStack are healthy before it starts the server, which exited when its first connection reached a Postgres still starting, andmake downalso stops the server of theserverprofile.- The local demo shows the
infra/instances ofexample-infrain its graph output, its table of the example and its scenarios, and no longer says the example's run workflow lacksrun-name. - A dispatched apply job, once bound, could report a result for a stack of a later wave that no dispatch carried yet, moving it to applied before its dependencies, and could record a named check on a stack the apply skips. The server now binds a job to a stack that no dispatch carries yet only through a dispatch of that stack's wave and environment, and to a skipped stack never. When a wave has several dispatches in one environment, it picks the one whose stacks the workflow run's jobs name, instead of the first unbound one, which could bind a job to a sibling chunk. A read of shared outputs binds a workflow run that no dispatch holds yet only when its jobs name a stack of the reader's dispatch. See OIDC binding.
- An
ignore_inferredentry qualified with the stack's own repository, such asacme/infra//stacks/prod/vpcinacme/infra, suppresses the inferredreads_stateedge to that stack, which it never matched before.
0.3.0 - 2026-10-01
Added
- The page
/setup/callbackhas a step to give the App a logo, linking to the App's settings on GitHub and to the logo, served at/setup/logo.pngin and out of setup mode. An App without a logo shows its owner's avatar on every comment and check, and GitHub sets the logo only in its settings page. See Logo. - An apply that succeeds gets a reply on the pull request, as a failed one already did, with what was applied and whether the orchestration locks were released or stay held until the merge.
- The runs in
GET /v1/overviewandGET /v1/repos/{owner}/{repo}/runscarrystack_count, the firststack_keysand the summed plansummaryof their stacks, and the web UI's run tables show them as Stacks and Changes columns next to each run's duration. The overview drops the Repository column when there is one repository, and an apply in progress shows its current wave. See Runs of a repository.
Changed
- An apply with a pull request gets its own comment, posted under the
stackorder applycommand, or after the merge inon_mergemode, and edited as the waves run, with the pending environment approvals and a row per stack. The sticky comment keeps showing the plans and lists the pull request's applies, instead of being rewritten into the apply's state above the command, and the replies to a finished apply link its comment. See Checks and comments.
Fixed
- The install and settings links on
/setup/callbackopen in a new tab, so following one before storing the credentials no longer leaves the only page that shows them. - The web UI no longer scrolls sideways at phone width: screen reader text stays inside scrolling tables, the status cards of the overview shrink below 320px, and the run picker of a repository's graph fits the screen.
0.2.0 - 2026-09-30
It works with stackorder/actions v1.0.0 or later; the env input and secret of the reusable workflows need v1.1.0.
Added
SECURITY.md: the supported versions and private vulnerability reporting through GitHub security advisories, linked from the README and the security model.- Terraform module: outputs
alb_arn,https_listener_arn,http_listener_arnandtarget_group_arn, so a caller can attach its own listener rules, and single sign-on in front of the UI throughoidc_authenticationorcognito_authentication. The HTTPS listener then authenticates browsers before forwarding, while rules at priorities 1 to 5 (the module reserves 1 to 99) let GitHub webhooks, Actions jobs and the CLI with bearer tokens, the health checks and Prometheus through without a login. - Terraform module:
alb_https_egress_cidrs, HTTPS egress from the load balancer security group to IPv4 CIDRs,0.0.0.0/0by default when authentication is set, since identity providers publish no fixed ranges. - Terraform module:
public_tasksruns the tasks in the public subnets with public IPs and, withcreate_vpc, creates no NAT gateways; the service security group still admits only the load balancer. For a single-user deployment the NAT gateway was the largest fixed cost after the database. - Terraform module:
verify_image(on by default) fails the plan with an explanation whenimage_tagcannot be pulled anonymously from ghcr.io, instead of ECS retrying the pull until the deployment times out, anddeployment_timeoutbounds how long an apply waits for a steady state. - Terraform module:
db_password_versionandsecrets_versionrotate the database password and rewrite the app and metrics token secrets, now that their values are not in state. - Terraform module: one task runs while the GitHub App inputs are unset, so there is a single setup token, and the examples export
log_group_name. STACKORDER_SETUP_TOKENsets the setup token instead of the generated one.- A
visibilityjob in the release workflow checks, from a runner without a registry login, that every pushed image tag can be pulled anonymously from ghcr.io, and the release checklist makes package visibility a step, since GitHub publishes a new package as private. A docs test checks that every version pin matches the latest release. - Provider credentials on the Workflows page: the
envinput andenvsecret of thestackorder/actionsreusable workflows, which needstackorder/actionsv1.1.0, carry credentials for providers besides AWS, such as a Cloudflare API token, asKEY=VALUElines orKEY<<DELIMITERmulti-line values. The page covers passing the secret by name, theENVenvironment secret that replaces it for jobs under that GitHub environment, read-only tokens for plans and write tokens only in gated environments, andephemeralfor a token passed as aTF_VAR_variable. Getting started, the Terraform on GitHub Actions guide, the actions reference, Stack instances and Security hardening link to it. - Security hardening explains GitHub's immutable OIDC subjects,
repo:<owner>@<id>/<repo>@<id>, used by repositories created, renamed or transferred after July 15, 2026. It shows how to read a repository's subject prefix withgh api repos/<owner>/<repo>/actions/oidc/customization/sub, and every page that shows a trust policy links to it. - Environments and authorization covers personal accounts and GitHub Free: which of the five layers still hold, that private repositories on GitHub Free have no environment secrets, and a recommended
stackorder.yamlfor a single owner withrequire_approvals: 0and noallowed_teams,four_eyesorrequire_codeowner_review. - The
stackorder.yamlreference has a Layouts section that sets the defaultstacks/**layout beside a flat one:stacks.discover: ["*"]withmodules.paths: ["modules/**", "*/modules/**"]. - Getting started shows the repository form of
gh variable set STACKORDER_SERVER_URLnext to the organization form, since a personal account has no organization variables.
Changed
/setupopens only with a one-time setup token, so whoever reaches a fresh server before its operator can no longer register the GitHub App under their own account. The server logssetup_urlwith the token atwarnlevel once at start-up;?token=is traded for a signed cookie and dropped from the URL before the page loads, and without it the page answers403. The token stops working on the instance that created an App, and a generated one changes at every start. Creating a replacement App withSTACKORDER_ALLOW_RESETUPneeds it too. See the setup token and the security model.- The Terraform module keeps secrets out of Terraform state:
github_app_private_key,github_webhook_secret,github_oauth_client_secret,session_keyandmetrics_tokenare ephemeral inputs, the generated database password, session key and metrics token come from ephemeralrandom_passwordresources, and the Secrets Manager versions and the database password use write-only attributes. The module therefore requires Terraform or OpenTofu 1.11, the AWS provider 6.50 and the random provider 3.7, and the CI matrix validates 1.11 and 1.14. A caller must supply the ephemeral inputs at apply time as well as plan time. The first apply after upgrading from v0.1.0 writes a new database password and rewrites the secrets, so the service rolls and, unlesssession_keyandmetrics_tokenare set, everyone signs in again and scrapers read a new token; theself-hostedexample now takesgithub_app_idandgithub_oauth_client_idas variables. See the upgrade notes. - The Terraform module's first-deploy steps take the setup URL with its token from the task logs, and the release checklist pushes the release commit and its tag together so the docs site never pins a tag that does not exist.
Fixed
- The README, the introduction, how it works and the design document no longer say the server only sees metadata: it also receives redacted plan text capped at 256 KB, and never cloud credentials or Terraform state.
- The Workflows page no longer says
secrets: inheritgives providers their credentials: the reusable workflows read no secret, and GitHub passes inherited secrets only to a reusable workflow in the caller's own organization or enterprise, so it passed nothing tostackorder/actions. The caller files in the README, the guides, the Workflows and Stack instances pages and the design document no longer pass it. - The getting-started trust policies no longer imply that every repository's subject starts with
repo:<owner>/<repo>. - The README, the introduction and getting started no longer presume a GitHub organization: a personal account can deploy the server, create the App and be a required reviewer.
- The design document no longer says the module's task egress always goes through NAT.
0.1.0 - 2026-09-30
The first release. It works with stackorder/actions v1.0.0 or later.
Added
Wire types and configuration
api/v1wire types shared by the CLI, the server, the UI and tooling, with JSON and YAML tags: stacks, edges, graphs, runs, run stacks, checks, matrix entries (with the checkoutsha,plan_run_idandartifact), plan summaries, audit entries and thev1.Errorcodes. Stacks, affected stacks, run stacks, stack details and matrix entries carry theirinstance, and a stack itswatch_paths.- Loading, defaulting and validation of
stackorder.yamland per-stack.stackorder.yaml, with unknown keys as errors and per-stack merging. - Stack instances. A key
path:instancenames one deployment of a directory, with its own state object, var files, environment variables, GitHub environment and apply role, and every instance is a stack of its own in the graph, affected sets, waves, locks, runs, checks, plan artifacts, drift and the UI. stacks.instances.from_var_filesinstackorder.yaml, a glob relative to each stack directory: every matching file declares an instance named by its base name up to the first., and is that instance's var file after the root and stackvar_files. A derived name that is not a valid instance name is an error; a matched file an explicitinstanceslist leaves out is a warning.instancesin.stackorder.yaml, a list of names or a map from name to overrides ofenvironment,workspace,backend_config,var_files,env,plan_output,apply.allowed_teams,depends_onandignore_inferred.backend_configat the root, per stack and per instance:-backend-configvalues forinit, asname=valueor a file relative to the repository root. The scanner overlays them on thebackend "s3"block, so inferredreads_stateedges and the shared-state warning work for partial backends and per instance.var_filesat the root, per stack and per instance:-var-filevalues forplan, the re-plan insideapplyanddrift, relative to the stack directory. A missing file is a scan warning and a plan error.envat the root, per stack and per instance: environment variables for the tool and the hooks, as a string or with aplan,applyanddriftvalue,driftfalling back toplan. Reserved names are refused, and values of secret-looking names are redacted.stacks.excludeinstackorder.yaml: directory globs that are never stacks, beatingstacks.discoverandstacks.include.environmentskeysprefix,prefix:instanceand:instance, the most specific key winning, with template values.- Go templates in
environmentsvalues,environment,workspace,backend_config,var_files,envvalues,depends_onandignore_inferred, with.Path,.Name,.Instanceand.Key, the functionstrimPrefix,trimSuffix,base,dir,replace,lowerandupper, the builtinsand,or,not,eq,ne,lt,le,gtandge, andifandwith. Other actions and builtins are refused, and output over 4096 bytes is an error, so a repository's configuration cannot stall the server. - The environment of a stack is the first of the instance's
environment, the stack's and theenvironmentsmatch that renders non-empty, then the instance name, thendefault. - Validation of the instance keys:
environmentsprefixes may not contain:, an empty prefix needs an instance part, and keys whose prefixes normalise to the same path with the same instance part are refused; a null item ininstancesor a nullenvvalue is refused; reservedenvnames and the instance namedefaultare refused in any letter case; twofrom_var_filesmatches that derive the same instance name are an error. depends_onentries with an instance suffix; a bare path to a directory with instances resolves to the instance of the same name.
CLI (stackorder)
- Commands
resolve,plan,apply,drift,check,graph,affected,unlockandversion, with exit codes 0, 1, 2 (changes or drift) and 3 (refused by the server or failed closed), GitHub Actions outputs and Markdown step summaries.--stacktakes a stack key,pathorpath:instance; a bare path on a directory with instances is an error that lists them. - Scanning of a checkout into the dependency graph: stack discovery,
s3backends, module sources normalised into graph identities, inferredreads_stateedges fromterraform_remote_state,depends_on(includingowner/repo//keyacross repositories),ignore_inferred, and changed paths from git with merge-base semantics. - Resolution of the affected set with reasons, propagation through modules, dependents and remote state, longest-path waves, cycle reporting, validation and Graphviz DOT output.
- Watch paths: the backend configuration files and var files outside a stack directory affect the stack with the reason
watch_path, and the tree hash covers*.tfvars,*.tfvars.jsonand*.tfbackendfiles. resolvefalls back to a local decision and marks itunconfirmedwhen the server cannot be reached, posting a neutral check withGITHUB_TOKEN.- Terraform and OpenTofu wrapper: tool detection,
initwith the stack'sbackend_configandSTACKORDER_BACKEND_CONFIG(and-reconfigurewhen a stack hasbackend_config, so instances of one directory can share a checkout), workspaces,plan,show,applyand plan JSON summaries. - Redaction of secrets from everything sent to the server, the step summary and fallback checks,
::add-mask::registration, and truncation of plan text. - Repository hooks
.stackorder/hooks/{pre-plan,post-plan,pre-apply,post-apply}.sh, run by the CLI in CI and locally withSTACKORDER_STACK,STACKORDER_RUN_ID,STACKORDER_PLAN_JSONandSTACKORDER_PLAN_FILE. The tool and the hooks also getSTACKORDER_STACK_PATHandSTACKORDER_INSTANCE. applyconfirms the run with the server, checks the run's SHA against the dispatchshaand the checkout'sHEAD, re-plans when the plan artifact is missing orapply.from_planis false, and applies only when the new plan's resource addresses match the recorded plan.apply --localfor manual applies under server locks with an API key;plananddriftstay local outside Actions unless a server, a run id and an API key are all set.- Server API client with retries, error mapping to sentinel errors and a fresh runner OIDC token for every call.
Server (stackorder-server)
- One
http.Servercomposing the webhook receiver, the queue workers, the scheduler, the JSON API and the embedded UI, configured from the environment, with thehealthcheckandversionsubcommands and graceful shutdown. - Setup mode when no GitHub App credentials are set:
/setupcreates the App from a manifest and prints its credentials once;STACKORDER_ALLOW_RESETUPallows creating another App later. - Run state machine for plan, apply and drift runs, superseding on a new pull request head, and roll-ups of per-stack status (
blocked,noop,unconfirmed,unknown,skipped). - Apply gate with five layers (requester and
allowed_teams, pull request state and approvals withfour_eyesandrequire_codeowner_review, fresh plans, named policy checks, locks), all failures reported together; policy read from the default branch. before_mergeapplies fromstackorder applycomments andon_mergeapplies on merge; comment commandsplan,applyandunlock, accepted from users with push permission and rate limited per pull request.- Stack locks taken all or nothing before wave 0 and released on merge or run completion, with daily reminders for closed pull requests that still hold locks.
- Dispatch of
stackorder-run.ymlper wave and GitHub environment, chunked byapply.max_parallel, binding of workflow runs to dispatches by run name and job names, and a reconciliation every minute that resends unsent dispatches, marks vanished jobsunknownand recovers apply runs left without a dispatch. - Cross-repository dependents: a merge that changes an upstream stack starts plan runs in the downstream repositories.
- Scheduled drift checks from
drift.schedule, one GitHub issue per drifted stack, closed when a later check finds no drift. - Pull requests are resolved under the
stackorder.yamlthey carry and under the default branch's, with the affected sets united. - Deployment protection rule answers for dispatched runs, and a neutral check for fork pull requests.
- Start-up and daily sync of installations and repositories from the App API, forgetting what GitHub no longer lists.
- Optional S3 artifact bucket for full plan text (
STACKORDER_ARTIFACT_BUCKET), served atGET /v1/runs/{id}/stacks/{key}/plan. - Retention pruning of plan text, events and drift history.
GitHub integration and security
- Hand-written GitHub App client: App JWT, cached installation tokens, retrying transport with rate limit handling and pagination, check runs, the sticky pull request comment, reviews, team membership, contents, workflow dispatch, runs, jobs and deployments, and OAuth.
- CODEOWNERS parsing with GitHub's matching semantics.
- Webhook HMAC verification, delivery deduplication and persistence.
- Verification of GitHub Actions OIDC tokens against the cached GitHub JWKS,
jtireplay protection, optionalSTACKORDER_REQUIRED_WORKFLOW_REFpinning, and binding of plan tokens to the pull request head and of dispatch tokens to their dispatch, attempt and environment. - Authentication of runners by OIDC token, automation by
sk_API keys and people by GitHub sign-in sessions, with same-origin checks on state-changing requests and per-organisation visibility. - Per-instance AWS roles and session names in
stackorder/actionsv1.0.0:aws-role-arn-mapkeyspath:instanceand:instance, matched before path prefixes, and theaws-role-session-nameinput on both reusable workflows, a name or a name per mode.
API, metrics and tracing
- Runner endpoints for runs, graph uploads, stack results, check verdicts and unlocks.
- Human and automation endpoints for the overview, repositories, graphs (by commit SHA, SHA prefix or
default), runs, stacks, modules and their consumers, unlocks, re-runs and the audit log, paginated with cursors. /healthz,/readyzand/metrics, the latter optionally behindSTACKORDER_METRICS_TOKEN.- Prometheus metrics with the
stackorder_prefix for runs, stacks, dispatches, drift, locks, commands, webhooks, the queue, GitHub requests and rate limit, HTTP requests and build info. - OpenTelemetry tracing over OTLP HTTP when
OTEL_EXPORTER_OTLP_ENDPOINTis set.
Storage
- Postgres schema through embedded golang-migrate migrations, run at start-up under an advisory lock.
- Queue of events and jobs claimed with
SKIP LOCKED, retries with backoff and dead letters; scheduler leader election by advisory lock. - Stable stack and module identities across graphs, module families and versions, and the default-branch graph of each repository.
Web UI
- Preact app embedded in the server with SPA fallback: overview, repositories, the dependency graph (dagre layout drawn as SVG), stack, run and module pages, GitHub sign-in, unlock and re-run actions, and the full plan text from the artifact bucket.
Terraform module (deploy/terraform)
- ECS Fargate service behind an Application Load Balancer with TLS, RDS PostgreSQL or Aurora Serverless v2, Secrets Manager secrets for the App credentials, and an optional artifact bucket.
- Container health check through the
healthchecksubcommand, a generated bearer token for/metrics, opt-in major database upgrades, optional load balancer access logs and WAF web ACL, and a 60 s drain before ECS stops a task. - Examples
complete,existing-vpcandself-hosted(Stackorder deploying itself), withterraform testsuites.
Packaging and release
- GoReleaser configuration for
stackorder_X.Y.Z_<os>_<arch>archives for linux, darwin and windows on amd64 and arm64, andstackorder_X.Y.Z_checksums.txt. - Multi-stage Dockerfile (Node for the UI, Go,
gcr.io/distroless/static:nonroot) with aHEALTHCHECK, published asghcr.io/stackorder/stackorder:X.Y.Z,:X.Yand:latestfor linux/amd64 and linux/arm64. - Makefile targets for building, the three test levels, linting, the UI, the docs site, a local development stack and the image; docker compose for Postgres and LocalStack.
Tests and CI
- Unit tests for every package, with
go test ./...needing no Docker, network or Postgres. - Integration tests (
integrationtag) running the server on Postgres, an in-memory fake GitHub API, a fake OIDC issuer and the CLI on a faketerraformthrough whole plan, apply, gate, drift, fork, manual apply, installation and cross-repo flows. - End-to-end tests (
e2etag) running real Terraform and OpenTofu against LocalStack S3 andstackorder/example-infra, and an optional live variant against a real GitHub organisation. - CI workflows for lint, unit, integration and build, the UI, the Terraform module, the docs site, nightly end-to-end runs and releases.
Documentation
- VitePress documentation site: guide (including getting started and a local demo), configuration (including Stack instances, with a worked example of one OIDC bootstrap role and a provider role per account, and a migration table from Terrateam), reference, operations, the design document with implementation notes, and the architecture contract.
ARCHITECTURE.mdandCONTRIBUTING.md.
Design deviations
Recorded in ARCHITECTURE.md and in the implementation notes of the design page on the documentation site.
- The CLI runs the repository hooks itself, in CI and locally, instead of the reusable workflow, and also sets
STACKORDER_PLAN_FILE;STACKORDER_PLAN_JSONandSTACKORDER_PLAN_FILEare file paths. stackorder-run.ymltakes a fifth input,sha, the commit every job checks out;modeisplan,applyordrift, neverresolve;stacksis required; the wrapper setsrun-name: stackorder ${{ inputs.mode }} ${{ inputs.run_id }} wave ${{ inputs.wave }}so the server can bind the workflow run to its dispatch.- Both reusable workflows require
server-url, and the calling jobs must carry the permissions block, since a called workflow cannot raise permissions. Theplan.ymlresolve job also needspull-requests: read. - Server-dispatched
plananddriftjobs run under the environmentdefaultand assume the plan role (aws-plan-role-arn, a newrun.ymlinput); onlyapplydispatches carry the stack's environment. Everyrun.ymljob runs underenvironment: ${{ matrix.environment }}in the concurrency groupstackorder-stack-<key>. A stack with no environment mapping runs underdefault, never an empty name. - The OIDC
shaclaim is never compared with the run's SHA. Plan runs are bound byevent_name, therefs/pull/<n>/mergeref and the pull request head read from GitHub; dispatched runs byworkflow_dispatch, the default branch ref, the dispatch'srun_idand the environment. apply.max_parallelcaps the number of stacks in one dispatch rather than concurrent jobs; the caller'smax-parallelinput limits jobs.pull_request_targetplans are not supported; fork pull requests get a neutral check from the server.- The CLI redacts with its own rules instead of a list of the runner's secret patterns.
- The implementation has more packages than the design names (
config,report,command,tf,client,principal,metrics,artifactsand others), more run and stack statuses (superseded,unconfirmed,unknown,skipped), themanualtrigger, and more API endpoints. - The schema stores a stack's backend as one
backendjsonb column and edges byfrom_keyandto_key, keeps a ref-less family row per module, and adds tables for graph membership, checks, dispatches, OIDCjtis and the audit log. - Named checks accept
warn, which passes; a stack'sapply.allowed_teamsreplaces the root list rather than narrowing it.