Skip to content

Troubleshooting ​

Stackorder degrades gracefully but never silently. Every refusal is a PR comment or a check with the reason, and every fallback is marked unconfirmed. Start from what you see.

You seeGo to
A neutral unconfirmed checkUnconfirmed checks
A comment refusing stackorder applyApply refused
stackorder/apply red, stacks blockedApply failed in a wave
Applies refused with locked after a PR was closedLocks after a closed PR
A stack stuck in unknownRunner died mid-apply
A wave finished but nothing happened nextLost webhooks
No reaction to a comment commandCommand ignored
stackorder/resolve failedDependency cycle
API errors unauthorized or forbidden in a job logOIDC rejected
A stackorder_outputs read refused, such as outputs_not_shared or state_location_unknownShared outputs refused
The server serves only /setupServer start-up
/setup answers 403 "Setup needs the setup token"Server start-up
The server exits at start saying the schema is dirtyRecover from a dirty migration

Where to look ​

SourceWhat it tells you
The check run output, the sticky PR comment and the apply's PR commentWhat the server decided, and why
The Actions job logWhat the CLI did, and the error it got from the server or from Terraform
The web UI run pageWaves, per-stack status, and links to job logs
The App's delivery log on GitHubWhether GitHub delivered a webhook, and the server's response
Server logsJSON by default; set STACKORDER_LOG_LEVEL=debug for more detail. Every request logs its request_id, which the response carries as X-Request-Id
The run's warnings in GET /v1/runs/{id}Dispatches GitHub refused, external dependents that were not scheduled
GET /v1/auditCommands ignored or rate limited, unlocks, config_invalid for a default-branch stackorder.yaml the server could not parse, outputs_read and outputs_refused for reads of a stack's outputs
GET /readyz and GET /metricsWhether the database is reachable; queue depth, webhook lag and rate limit. /metrics needs Authorization: Bearer <STACKORDER_METRICS_TOKEN> when the token is set, which the Terraform module always does

Unconfirmed checks ​

Symptom. Plans ran, but the checks are neutral and say unconfirmed. stackorder apply is refused.

Why. The CLI could not reach the server: a network error, a timeout, a 429 Too Many Requests from a rate limiter or reverse proxy in front of the server, or a 5xx answer, three retries later, or an empty server-url. It honours a Retry-After header on a 429 or 5xx, waiting up to a minute before each retry, and gives up at once when the header asks for longer. It computed the affected set locally, planned, and set a neutral check with the job's GITHUB_TOKEN, so nobody mistakes the fallback for a green light. Applies are refused because the server never confirmed the plans.

Fix.

  1. Check the server: curl -fsS https://stackorder.example.com/readyz.
  2. Read the resolve step in the job log. It shows whether the CLI ran in local mode or got an error from the server.
  3. If the CLI ran in local mode, the workflow passed an empty server-url: check that the STACKORDER_SERVER_URL Actions variable is set for the repository or its organization.
  4. A server that answers unauthorized or forbidden does not produce an unconfirmed check: the job fails. See OIDC rejected.
  5. Once the cause is fixed, re-plan: push to the PR or comment stackorder plan.

Apply refused ​

Symptom. A comment on the PR refuses stackorder apply, naming the failing layer and the reason. The CLI exits with code 3 when the server refuses.

The gate checks every layer and reports all failures in one comment:

RefusalFix
The commenter is not in a required team, or lacks push permissionAsk a member of every affected stack's allowed_teams, or apply a subset you are allowed to: stackorder apply stacks/staging/vpc.
Not enough approvals, or the PR is not mergeableGet the approvals apply.require_approvals asks for, on the head commit, from users with push permission; resolve conflicts.
No code-owner approval on the head SHAWith require_codeowner_review, an approval from the owning team must be on the current head commit. Re-request review after a push.
The requester is the PR authorWith four_eyes, someone else must comment stackorder apply.
A stack has no plan for the head SHAWait for the plans of the latest push, or fix the stack whose plan failed.
A named check failedFix what the check reported. warn does not block; fail does.
A stack is locked by another PRWait for that PR to merge, or see Locks after a closed PR.
Another apply of this PR is in flightWait for it to finish. An apply the server recorded but never dispatched is dispatched by the server within about three minutes, or, when its plans are no longer the head's, failed with its locks released and a comment. The locks stay held when an earlier apply of the PR changed what is deployed.
apply.mode is on_mergeMerge the PR; the merge starts the apply.

Gate policy comes from stackorder.yaml on the default branch. Changing it in the PR does not change the gate for that PR.

Apply failed in a wave ​

Symptom. One stack's apply job failed. stackorder/apply is red; its dependents are blocked; unrelated stacks in the same wave finished.

Why. A failed stack blocks every transitive dependent, and the run ends after the current wave. The PR keeps its locks, so nobody else can apply those stacks meanwhile.

Fix. Read the failed job's log, fix the code and push. The push invalidates the old plans and re-plans. Comment stackorder apply again; stacks that already applied plan as no-ops or small diffs, and the waves run from the start.

Locks after a closed PR ​

Symptom. A PR that applied was closed without merging. It got a warning comment, and other PRs touching the same stacks are refused with locked.

Why. After an apply, the deployed infrastructure matches the closed PR, not the default branch. The locks stay so nobody builds on a mismatch without noticing.

Fix.

  1. Decide how to bring the default branch and the deployment back in line: reopen and merge the PR, or open a new PR that changes the stacks back to what the default branch should deploy.

  2. Release the locks. With push permission, comment stackorder unlock on the closed PR, which releases its locks, or use the UI. With an API key, run the CLI:

    sh
    export STACKORDER_SERVER_URL=https://stackorder.example.com STACKORDER_API_KEY=sk_...
    stackorder unlock stacks/prod/vpc stacks/prod/apps --reason "PR 41 closed; reverting in PR 43"
  3. Apply the reconciling PR. Until then, a drift check reports the stacks as drifted.

Runner died mid-apply ​

Symptom. A stack is unknown in the run. Its orchestration lock is held, and the S3 state lock may be too.

Why. The job vanished before reporting a result: its workflow run completed without the stack reporting, or no workflow run picked the dispatch up within 30 minutes. If the stacks became unknown while the apply was still waiting for an environment's reviewers, with the warning workflow run not found, the wrapper lacks the run-name line and nothing was applied. Otherwise nobody knows how far apply got. Stackorder leaves the S3 lock as is; it never touches state locks. A result that arrives late still lands on an unknown stack.

Fix.

  1. Read what the job log shows of the apply.

  2. Check the state lock. With use_lockfile, a <key>.tflock object sits next to the state in the bucket; with DynamoDB, there is a lock item. If the lock is stale, release it from the stack directory with credentials that can write it:

    sh
    tofu force-unlock <LOCK_ID>

    The lock id is in the error any later plan prints while the lock is held.

  3. Release the orchestration lock. --force-state releases it like any unlock, and records in the audit log and in the PR comment that a runner died mid-apply and the state lock needs checking:

    sh
    stackorder unlock stacks/prod/vpc --force-state --reason "runner lost during apply"
  4. Re-plan the PR, push or comment stackorder plan, and apply again to converge.

Lost webhooks ​

Symptom. A wave's jobs finished but the next wave did not start straight away, or a check stayed pending.

Why. GitHub delayed or dropped a webhook. The server does not depend on it: every minute it reconciles open dispatches with GitHub, resending dispatches GitHub never accepted and binding workflow runs by their run-name, and the CLI's result call is the source of truth. A run moves on within about a minute even with no webhooks at all.

Fix. Usually none. If a comment command got no reaction, open the App's delivery log on GitHub. A delivery that failed can be redelivered; the server deduplicates by delivery id, so an event it already processed is not processed twice. Deliveries that fail the signature check point at a GITHUB_WEBHOOK_SECRET that does not match the App's.

Command ignored ​

The App adds an eyes reaction when it receives a command, and a rocket when it dispatches.

What you seeCause
No reactionThe webhook did not arrive (see above); the commenter has no push permission (audited as command_ignored); the command is inside a code block or a quote; or the first two words of the line are not stackorder and a known verb
A comment asking to waitMore than 10 commands in a minute on this PR
Eyes, no rocketThe command was refused, or needed no dispatch (help, unlock); look for the reply comment
Rocket on an apply, no reply yetThe apply is still running, or a wave is waiting for an environment approval. The apply comment under the command shows the current wave and links each pending approval, and the run details page lists every stack. The App replies when the apply succeeds or fails.
Rocket, but no apply commentEvery stack of the apply was a no-op, so it finished before its first update and the reply is its only comment; GitHub refused the dispatch, and the reply says the apply could not start (see the row below); or GitHub refused the comment, logged as update github for run, and the next update of the run posts it. A running apply whose comment was deleted, or whose comment the App can no longer edit after the App was replaced, is posted again on the next update; only a finished run keeps no comment.
Rocket, nothing in ActionsCheck that .github/workflows/stackorder-run.yml exists on the default branch and declares the inputs run_id, mode, wave, sha and stacks. A dispatch GitHub refuses is recorded as a warning on the run.

The command is the first line whose first word is stackorder and second word a known verb, both case insensitive.

Dependency cycle ​

Symptom. stackorder/resolve failed, and its output spells out the stacks that form a cycle.

Fix. Remove one depends_on entry in the cycle. If an inferred reads_state edge closes the cycle, suppress it with ignore_inferred in the reading stack's .stackorder.yaml. Check locally with stackorder graph.

Plans invalidated by a push ​

Symptom. stackorder apply is refused right after a push.

Why. Plans belong to a head SHA. A new push invalidates them, supersedes the run, and re-plans.

Fix. Wait for the new plans, then comment stackorder apply again.

Expired plan artifacts ​

Symptom. An apply re-planned instead of using the saved plan, then refused.

Why. With apply.from_plan: true, the CLI applies the saved plan file. When the artifact has expired it re-plans, and refuses unless the new plan touches the same resource addresses as the recorded one.

Fix. Re-plan the PR (push or comment stackorder plan), review the new plan, and apply again. The reusable plan.yml keeps plan artifacts for 5 days, so an apply requested later than that always meets an expired artifact; the repository's artifact retention setting cannot extend it.

OIDC rejected ​

Symptom. A job log shows the server answering unauthorized or forbidden.

CauseAnswerFix
The job has no id-token: writeThe CLI cannot get a tokenGrant it on the job that calls the reusable workflow. A called workflow cannot widen its caller's permissions.
Wrong audience401The CLI requests the server-url it was given as audience, with any trailing slash removed; it must equal STACKORDER_OIDC_AUDIENCE, which defaults to STACKORDER_BASE_URL without a trailing slash.
Token too old, or reused401Tokens issued more than 10 minutes ago and tokens already seen are refused. The CLI requests a new one for every call, so this points at a clock problem or a replay.
Repository unknown or suspended403Install the App on the repository, or unsuspend the installation. The server learns installations from webhooks, at start-up and daily.
job_workflow_ref does not match403With STACKORDER_REQUIRED_WORKFLOW_REF set, only the pinned reusable workflows can register runs and post results.
Wrong environment403A dispatched apply job must run under the stack's environment, and a plan or drift job under default.
Re-running a completed workflow run403A re-run of a workflow run whose dispatch already completed is refused. Start a new run: push, comment stackorder plan or stackorder apply.
A superseded run409 supersededA newer commit replaced the run; its jobs' results are refused.
A finished plan run409 conflictOnce a pull request's plan run is planned or failed, its pull_request jobs can no longer post results or check verdicts. Post verdicts from a post-plan.sh hook, before the result. "Re-run failed jobs" reports to the finished run; re-run all jobs, push or comment stackorder plan to plan again.
Fork pull requestForks get no id-token permission. Stackorder posts a neutral check and runs nothing.

Shared outputs refused ​

Symptom. A plan or apply fails at a data or ephemeral stackorder_outputs block, with an error code from the server. See Shared outputs.

The reading stack is the consumer and the stack it names is the producer. Refusals of the caller, the run, the producer or its policy, state_location_unknown included, and a state location that no grant of the producer's repository covers, state_bucket_not_allowed, are in the audit log as outputs_refused, with the code as details.reason. outputs_disabled and the other codes of the state read are not.

CodeCauseFix
outputs_disabled (404)The server reads no state: STACKORDER_STATE_BUCKETS is not set.Set it to owner/repo=bucket/prefix entries for the buckets and prefixes that hold each producer repository's state, with shared_outputs_state in the Terraform module, and restart the server.
outputs_not_shared (403)The producer's default-branch outputs.readers does not list the consumer, or the producer's outputs configuration does not parse or render.Add the consumer's key, or a pattern that matches it, to outputs.readers in the producer's .stackorder.yaml, its instance override or stackorder.yaml, and merge it: a pull request's copy does not count. Check that a more specific level does not set its own readers, such as readers: []. An instance needs its suffix, as in stacks/prod/apps:blue or stacks/prod/apps:*.
not_found (404)The producer is not a stack its repository's default branch still has: stackorder.yaml does not discover its directory, the directory was deleted on the default branch, or the instance is no longer declared, or no longer has its var file. Or the key or repository is misspelled, or the repository is suspended or in another installation. For a consumer in another repository, also a producer that does not list it, or whose outputs configuration does not resolve.Check the stack attribute, and merge the producer before reading it. Check that stackorder.yaml on the default branch discovers it and that the instance is declared there, or has its var file. List a consumer of another repository with its repository, as acme/infra//stacks/prod/eks. Both repositories must be covered by the same App installation.
state_location_unknown (409)No apply or drift job of the producer has reported where its state is: none has run on a CLI that reports it, the producer's backend is not s3 with a bucket and a key, so no job reports its state object, or the last object reported lay outside the entries of STACKORDER_STATE_BUCKETS for the producer's repository, which the server does not record: it clears the location it had, logs the object at warn level, and says so in the message.Run an apply or drift job of the producer with a CLI that reports the state object: its next apply, a scheduled drift check, which records the location without changing anything, or stackorder apply --local. A producer whose backend is not s3 cannot share its outputs. If the server logged the object as not granted, first add an entry for the producer's repository that covers it, then run one of those jobs.
state_bucket_not_allowed (403)The producer's recorded state object is outside every entry of STACKORDER_STATE_BUCKETS granted to the producer's repository or to *. The server records only a location the entries cover, so the entries or the repository's name changed since, or shared outputs were off when it was recorded: its bucket or key is not listed, the location is granted to another repository, or the key only starts like a prefix without being under it, as stacks/production/ for stacks/prod. The server logs the repository, the stack, the bucket and the key at warn level.Add an entry for the producer's repository, owner/repo=bucket/prefix, whose bucket and prefix cover the key, to STACKORDER_STATE_BUCKETS, and grant the server read access to it. If the location belongs to another repository's state, the producer's backend is pointed at the wrong object: fix the backend and run an apply or drift job of the producer.
state_access_denied (403)S3, or KMS through S3, refused the server's credentials, or STS refused to let the server assume STACKORDER_STATE_ROLE_ARN. Without s3:ListBucket on the prefix, S3 answers a request for a missing object this way too.Grant s3:GetObject on the objects, s3:ListBucket limited to the prefix and kms:Decrypt on the bucket's KMS key to the task role, or to STACKORDER_STATE_ROLE_ARN; a bucket or key in another account must also allow that role in its policy. With a role, check that its trust policy admits the server's task role and that the task role has sts:AssumeRole on it, which the module grants with shared_outputs_role_arn. When the trust policy requires an external ID, set STACKORDER_STATE_EXTERNAL_ID (shared_outputs_external_id) to the value its sts:ExternalId condition expects. With the s3:ListBucket grant in place, a missing object answers state_not_found instead, which tells the two causes apart.
state_not_found (404)The object, or its bucket, does not exist where the producer's last apply or drift job found it: the backend key changed, or the state was moved or deleted.Run an apply or drift job of the producer so that it reports where its state is now.
state_encrypted (422)OpenTofu encrypts the producer's state.The server never decrypts state, so shared outputs cannot read this stack; keep terraform_remote_state for it.
state_unsupported (422)The producer's state has a format version other than 4.Apply the producer with a Terraform or OpenTofu release that writes version 4 state.
state_too_large (422)More than 64 MiB of the state precede its outputs, or the outputs, or a single string or number before them, exceed 4 MiB.Move large values out of the producer's outputs, or split the stack.
state_unreadable (422)The object is not valid state JSON, its recorded region is not a valid region, or S3 refused the request for another reason.Check that the object is the producer's state and that its backend's region is right, then run an apply or drift job of the producer.
unavailable (503)S3 throttled the read or answered with a server error, the read waited more than 20 s for one of the server's 4 read slots, or it did not finish within 30 s of getting one. Or the server could not write the read's outputs_read audit row, so it served nothing.Retry. The server logs the S3 error, never state content. For the audit row, it logs record audit at error level with the database error: check the database, then retry.
forbidden (403)The token does not prove the consumer: the run id or the stack key is missing or wrong, the stack is not one of the run's, its job already reported, or a dispatched job's stack has not been dispatched.Read from the consumer's own Stackorder plan, apply or drift job, where the CLI sets STACKORDER_RUN_ID and STACKORDER_STACK.
conflict (409), superseded (409)The run finished, a pull request's plan run is already planned, or a newer commit superseded the run.Plan again: push, or comment stackorder plan.

Server start-up ​

SymptomCause
The server exits at startThe log lists every invalid variable: DATABASE_URL or STACKORDER_BASE_URL missing, only some of the three GITHUB_APP_* variables set, a malformed private key, STACKORDER_ARTIFACT_PREFIX without a bucket, an entry of STACKORDER_STATE_BUCKETS without a repository or a valid bucket name, or with a . or .. segment in its prefix, a STACKORDER_STATE_ROLE_ARN that is not a role ARN or is set without STACKORDER_STATE_BUCKETS, a bad duration or pattern. Or the database cannot be reached or migrated.
The server exits at start saying the schema is dirty at a versionA migration failed on an earlier start or migrate run and left the schema marked unfinished. The server refuses to start a half-migrated database. See Recover from a dirty migration.
Only /setup, /healthz and /readyz respondSetup mode: the GitHub App variables are not set.
/setup answers 403 "Setup needs the setup token"/setup needs the one-time setup token. Open the setup_url from the server's log, in the same browser for the whole setup. A generated token changes at every start, so take the latest line; with two instances, each has its own token. With STACKORDER_LOG_LEVEL=error the line is not logged: use info or warn, or set STACKORDER_SETUP_TOKEN.
/setup says the server already has a GitHub App, but it runs in setup modeAn App was already created with this start's token. Restart the server with that App's credentials; if they were lost, delete the App in its GitHub settings and restart the server for a new token.
/healthz passes, /readyz failsThe database is unreachable.
Everyone was signed outSTACKORDER_SESSION_KEY changed, or differs between instances.

Recover from a dirty migration ​

Symptom. A server exits at start, or stackorder-server migrate exits 1, with an error like:

text
schema is dirty at version 7: a previous migration did not finish and left the schema marked unfinished

stackorder-server migrate --status prints dirty=true and exits 1.

Why. Migrations record their progress in a schema_migrations table: version N is marked dirty before migration N runs and the flag is cleared when it finishes. A migration that fails — a statement errored, or the process was killed mid-migration — leaves the flag set. Every later migration and every server start then refuse the database until the flag is cleared, so a half-migrated schema is never served.

What state the database is in. Each migration file runs in a single transaction. A migration that fails rolls its own changes back, so the schema is left at the version before the one that failed, and only the dirty flag remains. Confirm nothing from the failed migration survived before clearing the flag; a migration killed by a crash rolls back the same way.

Fix.

  1. Take a backup (or a snapshot) before touching the database.

  2. Read the error for the dirty version N and find migration N in the server's logs or the release notes to see what it was doing.

  3. Check the schema. Because migration N runs in one transaction, a failed N leaves the schema at version N-1, with none of N's objects present. Verify that is so — look for the tables, columns or indexes N would have added — and, if a crash left anything behind, drop it so the schema matches version N-1 exactly.

  4. Record the recovered version and clear the flag with the new image's migrate --force, against the same database:

    sh
    docker run --rm -e DATABASE_URL="$DATABASE_URL" \
      ghcr.io/stackorder/stackorder:X.Y.Z migrate --force N-1

    Pass the version the schema is actually at. After a rolled-back migration N that is N-1; migrate --force 0 records "no migration applied", for a failed first migration. Force version N itself only if you finished migration N by hand instead of reverting it. --force changes nothing but the bookkeeping row, and refuses a database that is not dirty.

  5. Migrate the rest and start the server:

    sh
    docker run --rm -e DATABASE_URL="$DATABASE_URL" \
      ghcr.io/stackorder/stackorder:X.Y.Z migrate

    With --force N-1, this re-applies migration N and any later ones; migrate --status should then print dirty=false and the current version. Start the server as usual.

Fix the cause of the original failure — commonly data the migration could not transform, or a disk or connection lost mid-run — before the next attempt, or N fails and marks the schema dirty again.