Skip to content

Architecture ​

This file is the contract between the packages of this repository and the stackorder/actions, stackorder/terraform-provider-stackorder and stackorder/example-infra repositories. The design document is the source of truth for behaviour; this file pins the shapes, names and library choices that let packages be built independently and still fit together, and records where the implementation deliberately departs from the design. Change it in the same commit as the code it describes.

Repositories ​

RepositoryContents
stackorder/stackorderGo module github.com/stackorder/stackorder: server, CLI, embedded UI, migrations, docs site, Terraform deployment module
stackorder/actionssetup JavaScript action, resolve / plan / apply / drift composite actions, reusable plan.yml and run.yml
stackorder/terraform-provider-stackorderTerraform and OpenTofu provider stackorder/stackorder: the stackorder_outputs data source and ephemeral resource; imports api/v1
stackorder/example-infraDemo monorepo used by the end-to-end tests

Layout of this repository ​

text
api/v1/                    wire types shared by CLI, server, UI and tooling (JSON + YAML tags, no logic)
cmd/stackorder/            CLI main; binds internal/scan, internal/graph and internal/validate into internal/cli
cmd/stackorder-server/     server main; wiring only, plus the `migrate`, `healthcheck` and `api-key` (create/list/revoke with optional read-only, installation and expiry scope; needs DATABASE_URL) subcommands
internal/config/           stackorder.yaml and .stackorder.yaml: defaults, validation, per-stack merge
internal/scan/             HCL scanning: discover stacks, parse backends, module sources, remote state, stackorder_outputs reads; git diff
internal/graph/            pure resolution algorithm: affected set, propagation, waves, cycles, DOT output
internal/validate/         composes config, scan and graph into the `validate` command's diagnostics report
internal/report/           pure rendering of check-run output, the sticky PR comment, refusals, drift issues
internal/command/          parsing of `stackorder …` PR comment commands
internal/tf/               terraform / tofu process wrapper, plan JSON summary, metadata-only resource list from state, redaction, hooks
internal/client/           HTTP client for the server API used by the CLI, incl. runner OIDC token fetch
internal/cli/              cobra commands: resolve, plan, apply, drift, check, graph, affected, validate, unlock, version
internal/store/            pgx queries; migrations embedded from /migrations (package `migrations`, `migrations.FS`)
internal/gh/               GitHub App client: JWT, installation tokens, checks, comments, dispatch, reviews, teams
internal/gh/codeowners/    CODEOWNERS parsing with GitHub's matching semantics
internal/oidc/             verification of GitHub Actions OIDC tokens and claim binding
internal/principal/        caller identity and the error vocabulary shared by runs and api
internal/runs/             run state machine, apply gate, locks, wave dispatch, superseding, drift, reconciliation, shared outputs, resource lists
internal/webhook/          HMAC verification, delivery dedup, event persistence
internal/worker/           queue: claim events and jobs with SKIP LOCKED, retries, idempotent handlers
internal/sched/            cron scheduler for drift and housekeeping; leader by advisory lock
internal/metrics/          Prometheus registry, all metric names, HTTP and GitHub client instrumentation
internal/api/              HTTP handlers, auth middleware, sessions and OAuth, API keys, /setup manifest flow
internal/ui/               embed.FS serving of the built UI with SPA fallback
internal/server/           composition of all of the above into one http.Server; used by cmd and by tests
internal/artifacts/        optional S3 store for full plan text (STACKORDER_ARTIFACT_BUCKET); Put, Get and Delete
internal/stateread/        reads only the root outputs of a state object a grant allows the producer's repository (STACKORDER_STATE_BUCKETS); stores and logs nothing
internal/testutil/pgtest/  Postgres for tests: TEST_DATABASE_URL or testcontainers, one database per test
internal/testutil/ghfake/  in-memory fake of the GitHub API with inspection and payload builders
internal/testutil/oidcfake/ fake OIDC issuer and runner token endpoint
internal/testutil/faketf/  fake terraform and tofu binary answering per stack from a JSON config, for CLI end-to-end tests
migrations/                golang-migrate SQL files, NNNN_name.up.sql / .down.sql, embedded as migrations.FS
ui/                        Preact + Vite + TypeScript app; `npm run build` writes internal/ui/dist
docs/                      VitePress documentation site
deploy/terraform/          ECS Fargate + RDS + ALB module; examples/ and tests/
test/integration/          server + Postgres + fake GitHub + the CLI on faketf, whole-flow tests (build tag `integration`)
test/e2e/                  real terraform + LocalStack + example-infra + server (build tag `e2e`)
Dockerfile                 multi-stage: node (ui) -> go -> gcr.io/distroless/static:nonroot
.goreleaser.yaml           CLI releases for linux/darwin/windows, amd64/arm64, checksums; server image
docker-compose.yml         local Postgres and LocalStack for development and tests

Dependency direction: api/v1 <- config <- scan, graph, command, report, tf <- client, cli (runner side); internal/validate composes config, scan and graph, and cmd/stackorder binds it into cli with the scanner and the graph, so cli imports none of the three; api/v1 <- store, gh, oidc <- principal <- runs <- webhook, worker, sched, api <- server <- cmd; runs also imports stateread for its Location and State types. runs talks to GitHub through its GitHub interface (Client, ListInstallations, InstallationRepos), satisfied by *gh.App. Nothing under internal/ imports internal/cli or internal/server except cmd and tests. internal/graph, internal/report and internal/command import nothing but api/v1, internal/config and the standard library (plus the YAML, glob and cron libraries config pulls in). internal/runs never imports internal/api and vice versa; they share internal/principal.

Libraries ​

These are fixed. Do not introduce an alternative for the same job.

JobLibrary
HTTP routingnet/http ServeMux with method and wildcard patterns ("POST /v1/runs/{id}/graph")
Postgresgithub.com/jackc/pgx/v5 with pgxpool; no ORM, no sqlx
Migrationsgithub.com/golang-migrate/migrate/v4 with the pgx/v5 database driver and the iofs source over an embedded FS
YAMLgopkg.in/yaml.v3
HCLgithub.com/hashicorp/hcl/v2 (hclparse, hclsyntax), github.com/zclconf/go-cty for module argument values, and github.com/hashicorp/terraform-config-inspect/tfconfig
Plan JSONgithub.com/hashicorp/terraform-json
Globsgithub.com/bmatcuk/doublestar/v4
JWTgithub.com/golang-jwt/jwt/v5 (App JWT signing and OIDC token parsing)
Crongithub.com/robfig/cron/v3 parser and scheduler
CLIgithub.com/spf13/cobra
Metricsgithub.com/prometheus/client_golang
Tracinggo.opentelemetry.io/otel with the OTLP HTTP exporter, enabled only when OTEL_EXPORTER_OTLP_ENDPOINT is set
S3 (optional artifact bucket and state reads for shared outputs)github.com/aws/aws-sdk-go-v2 service/s3 and config
Assumed role for state readsgithub.com/aws/aws-sdk-go-v2 service/sts and credentials/stscreds
Logginglog/slog. Server: JSON handler unless STACKORDER_LOG_FORMAT=text. CLI: text handler unless STACKORDER_LOG_FORMAT=json
IDsgithub.com/google/uuid
Testsstandard testing, github.com/stretchr/testify/require and assert, github.com/google/go-cmp/cmp for diffs
Postgres in testsgithub.com/testcontainers/testcontainers-go/modules/postgres, skipped when TEST_DATABASE_URL is set and used directly instead
GitHub APIhand written in internal/gh on net/http; no go-github
UIPreact, TypeScript, Vite, preact-iso router, @dagrejs/dagre layout with hand written SVG, Vitest and Testing Library, Playwright for browser tests
DocsVitePress, vitepress-plugin-mermaid with mermaid 11
Actions repo@actions/core, @actions/tool-cache, @actions/http-client, esbuild, TypeScript, ESLint with typescript-eslint, Vitest
Terraform module providershashicorp/aws ~> 6.0, hashicorp/random ~> 3.6, hashicorp/http ~> 3.4 (webhook IP ranges only)

Go version is the one in go.mod; GOTOOLCHAIN=auto downloads it. Node 24. go.mod carries ignore node_modules so ./... skips the UI and docs dependencies.

Identities and names ​

  • Stack key inside a repo: path or path:instance (v1.StackKey(path, instance), v1.SplitStackKey). Paths are repository relative, slash separated, no ./, no trailing /. The suffix names an instance of the directory (see Stack instances); it is never, by itself, a Terraform workspace. A stack that sets workspace: blue and declares no instances is one instance named blue, keyed path:blue, whose Terraform workspace is blue.
  • Qualified stack key: owner/repo//key. A cross-repo depends_on target, or the stack of a stackorder_outputs block in another repository, becomes a v1.Stack with External: true, Repo set, Path and Instance split out, and Key set to the qualified key, so it can never collide with a local stack; the edge's To.Key is that qualified key. External stacks are never scheduled.
  • Module keys: see api/v1/doc.go. uses_module edge Meta always has ref (empty for local modules) and source (the raw spelling at that call). The store keeps a ref-less family row per module (modules.base_key) that module versions attach to; v1.ModuleDetail.Key and v1.ModuleConsume.ModuleKey are the family key without @ref, while graph nodes and v1.ModuleConsumer.Ref carry the pinned ref.
  • Inferred reads_state edges have Inferred: true. Those inferred from a terraform_remote_state data source carry Meta bucket and key (the full S3 object key, workspace prefix included); those inferred from a stackorder_outputs data source or ephemeral resource carry Metasource: stackorder_outputs and may point at an external stack. When both blocks infer the same edge, the remote state Meta is kept. A stack's ignore_inferred suppresses the edge in the scanner with a warning; an explicit depends_on to the same target replaces it.
  • Resolution rules the graph package fixes: a changed path belongs to the deepest enclosing stack directory only, and affects every workspace of that directory; paths under .terraform are ignored; git and registry modules never match changed paths; edge direction is From depends on To, so To applies first and wave(To) < wave(From).
  • Server side primary keys are UUIDs (runs.id, stacks.id, modules.id); GitHub ids (installations.id, repos.id) are the GitHub numeric ids.
  • Run ids appear in URLs, workflow inputs and check run output as the UUID string.
  • Check run names: stackorder/resolve, stackorder/plan, stackorder/plan: <key>, stackorder/apply, stackorder/apply: <key>, stackorder/<check-name>: <key> for named policy checks.
  • Sticky PR comment: one per PR, found by the hidden marker <!-- stackorder:sticky --> on its first line (and by the App's own login).
  • Check runs (runs.check_runs) and an apply run's comment (runs.comment_id) record ids of GitHub objects the App created. When editing one is refused because it is gone or no longer the App's to edit — a 404, a 410 or a non–rate-limit 403, which gh.ErrEditRefused matches and GitHub returns after the App of a deployment is replaced — the server creates a fresh one rather than give up; a rate limit or a 5xx is transient and never does. A comment_id of 0 marks a terminal run that has no comment and gets none, not a live run whose comment is suppressed.
  • Plan artifact name: v1.PlanArtifactName(key, sha), i.e. stackorder-plan-<slug>-<sha>, where v1.StackKeySlug(key) is the key with / and : replaced by -, then - and the first 8 hex characters of sha256(key), so keys such as a/b and a-b never share a name (the design's name has no hash suffix). The plan file inside it is <artifact name>.tfplan, written under STACKORDER_PLAN_DIR (default .stackorder/plans) and reported as an absolute path.
  • Workflow files in user repos: .github/workflows/stackorder-plan.yml and .github/workflows/stackorder-run.yml (id stackorder-run.yml is what the server dispatches). stackorder-run.yml inputs: run_id, mode (plan, apply or drift), wave, sha (commit to check out) and stacks (JSON array of v1.MatrixEntry). The wrapper sets run-name: stackorder ${{ inputs.mode }} ${{ inputs.run_id }} wave ${{ inputs.wave }} so the server can correlate the workflow_run event with its dispatch.
  • Stacks with no environment mapping run under v1.DefaultEnvironment (default), never under an empty environment name. Server-dispatched plan and drift jobs always run under default and assume the plan role; only apply dispatches carry the stack's environment.
  • Hooks .stackorder/hooks/{pre-plan,post-plan,pre-apply,post-apply}.sh are run by the CLI itself, in CI and locally, with STACKORDER_STACK, STACKORDER_STACK_PATH, STACKORDER_INSTANCE, STACKORDER_RUN_ID, STACKORDER_SERVER_URL, STACKORDER_REPOSITORY, STACKORDER_PLAN_JSON and STACKORDER_PLAN_FILE (both file paths) set, plus the stack's configured env variables for the mode (see Stack instances). The design assigns hooks to the reusable workflow; the CLI owns them so local and CI behaviour match. A hook that exits non-zero fails its job: a pre-plan, post-plan or pre-apply hook fails before or instead of the tool, and a post-apply hook fails the stack after a successful apply, reporting v1.ResultFailure with the apply's resources and state still attached so the server blocks the stack's dependents (the infrastructure did change).
  • Release assets on stackorder/stackorder tags vX.Y.Z: stackorder_X.Y.Z_<os>_<arch>.tar.gz (.zip on windows) containing the stackorder binary, os in linux, darwin, windows, arch in amd64, arm64, plus stackorder_X.Y.Z_checksums.txt (sha256). The server image is ghcr.io/stackorder/stackorder:X.Y.Z and :latest.

Stack instances ​

A stack directory can be deployed several times, once per instance: the same code with its own state object, var files, environment variables and GitHub environment. Every instance is a stack of its own everywhere a stack appears: the graph, affected sets, waves, locks, runs, checks, artifacts, drift and the UI. Nothing below needs the server to read the repository; the CLI renders everything from the checkout and the server re-resolves policy from the default branch with the same pure functions.

Identity ​

  • v1.Stack.Instance is the key suffix. v1.Stack.Workspace is the Terraform workspace the CLI selects after init; empty means none. v1.RunStack, v1.AffectedStack, v1.StackDetail and v1.MatrixEntry carry instance next to workspace. v1.Stack.WatchPaths lists repository-relative files that the stack reads at init or plan (backend config files, var files) and that its own directory does not own under the deepest-directory rule, so a file inside a nested stack directory is a watch path of the enclosing stack that reads it.
  • Compatibility: an object whose Instance is empty but whose Workspace is set came from a CLI older than this contract; readers treat the workspace as the instance. Nothing derives Workspace from a key any more (graph/resolve.go, graph/matrix.go, graph/validate.go, runs/dispatch.go, runs/resolve.go, scan.go, store/graphs.go use Instance).
  • Instance names match [A-Za-z0-9][A-Za-z0-9._-]*, are at most 64 characters, and are never default. Job-name matching, comment parsing, artifact names and GitHub environment names all tolerate that alphabet.
  • The store adds no column: stacks.key carries the instance and store.Stack.ToV1 splits it out; stacks.workspace stays the Terraform workspace.

Configuration ​

Root stackorder.yaml has these instance keys:

yaml
stacks:
  exclude: ["infra/state-backend"]              # directory globs, never stacks
  instances:
    from_var_files: "workspaces/*.tfvars.json"  # stack-relative glob
backend_config:                                 # -backend-config values for init; at the root only
  - infra/state.s3.tfbackend                    # a file, repository relative        when every stack
  - 'key={{ trimPrefix "infra/" .Path }}/{{ .Instance }}.tfstate'  #                 has a partial backend
var_files: []                                   # -var-file values for plan, stack relative
env:                                            # for the tool and the hooks
  TF_VAR_environment: "{{ .Instance }}"
  TF_VAR_role: { plan: reader, apply: deployer }  # per mode; drift falls back to plan
environments:
  "infra/": "infra-{{ .Instance }}"             # prefix, prefix:instance or :instance

.stackorder.yaml has instances, backend_config, var_files and env. instances is a list of names or a map from name to overrides; JSON always uses the map form (v1.StackConfig.Instances, custom (un)marshalling). An override (v1.InstanceConfig) may set environment, workspace, backend_config, var_files, env, plan_output, apply.allowed_teams, depends_on, ignore_inferred and outputs. workspace on a stack with instances is a template such as "{{ .Instance }}".

  • The instance set of a directory (config.InstanceNames(root, path, stack, matchedVarFiles) ([]string, error), with matchedVarFiles from config.MatchVarFiles(root, dir) on disk, or config.MatchVarFilesFS(root, fsys) over any fs.FS) is, in order of precedence: the names in instances; else one per file matched by from_var_files, named by the file's base name up to its first ., sorted; else [workspace] when the stack's rendered workspace is neither empty nor default (the single workspace instance; the template renders with an empty .Instance); else one instance with an empty name and a suffix-less key. A derived name that is not a valid instance name (default.tfvars.json, say), two matched files that derive the same name, and a stack workspace that renders to an invalid instance name are errors naming the file; a matched file whose derived name is not in an explicit instances list is a warning (config.VarFileWarnings) and is not used. Matches are ignored when from_var_files is unset.
  • Var files of an instance, in order: root var_files, stack var_files, the matched from_var_files file whose derived name equals the instance name, if any, then instance var_files, so that a later, more specific file wins in Terraform. Paths are relative to the stack directory. backend_config concatenates root, stack, instance; a value containing = is name=value, anything else is a file path relative to the repository root. env merges per variable name, instance over stack over root; a value is a string or an object with plan, apply and drift keys; drift falls back to plan, and a mode with no value leaves the variable as the job already has it. The re-plan inside apply runs with the apply values. depends_on and ignore_inferred concatenate stack and instance lists. workspace, plan_output, tool and tool_version take the most specific value set; apply.allowed_teams takes the most specific non-empty list.
  • outputs (v1.OutputsConfig) names the stacks that may read the root module outputs of a stack, in stackorder.yaml, in .stackorder.yaml and on an instance: readers lists them and sensitive (share or withhold, default withhold) says whether sensitive outputs are served. A reader is a doublestar pattern over the stack keys of the same repository (stacks/prod/eks, stacks/prod/apps:*, **), or owner/repo//pattern for stacks of another repository of the same installation, where owner/repo is literal; * does not cross /, ** crosses / only as a whole path segment (stacks/**/*:blue matches every blue instance under stacks, stacks/**:blue only those one level down), : is an ordinary character, and repositories compare without regard to case. A pattern must be spelled as a stack key is: no surrounding space, no ./, //, .. or trailing /, and no :default or empty instance, else validation names the spelling to write; an entry with a template is checked once rendered. readers comes from the most specific level that sets it, instance over stack over root, and replaces the others, unlike apply.allowed_teams: readers: [] on a stack or an instance withdraws a root grant, and unset at every level means the outputs are not shared. sensitive takes the most specific value set, and anything but share resolves to withhold. A null outputs, a null readers or a null reader is an error, so none of them can pass for unset or for no reader, and v1.OutputsConfig.Readers has no omitempty, so nil (null) and [] stay apart in JSON and in repos.config. Effective.OutputReaders (rendered; nil when unset) and Effective.OutputSensitive (share or withhold) carry the result, and config.ReaderAllowed(readers, producerRepo, readerRepo, readerKey) is the one matcher: a same-repository entry matches only readers of the producer's repository, and an invalid entry never matches. The warning of a pull request whose stackorder.yaml differs names outputs by sub-key, as it does apply, also when the block is added or removed whole.
  • Templates are Go text/template with missingkey=error, data .Path, .Name (the last path segment), .Instance and .Key, and functions trimPrefix prefix s, trimSuffix suffix s, base, dir, replace old new s, lower and upper, plus the builtins and, or, not, eq, ne, lt, le, gt and ge, and if and with; range, define, template, block and every other builtin are refused, and a template or function whose output exceeds config.MaxRenderedLength (4096 bytes) fails, so a configuration value cannot stall the server. They apply to environments values, environment, workspace, backend_config, var_files, env values, depends_on, ignore_inferred and outputs.readers. A string without {{ is used as is. A rendering error is a validation error of the file that holds the template. config.Render is the one implementation.
  • environments keys are prefix, prefix:instance or :instance. A key matches when its prefix matches the path (whole segments; an empty prefix, allowed only with an instance part, matches every path) and, when it has an instance part, that part equals the instance. The most specific key wins: one with an instance part before one without, then the longest prefix. The instance part must be a valid instance name; a prefix may not contain :, and two keys whose prefixes normalise to the same path (infra, infra/, ./infra) with the same instance part are an error. (The actions' aws-role-arn-map deliberately has no prefix:instance form; an exact path:instance key covers that case there.)
  • The GitHub environment of a stack is, in order: the instance override, the stack's environment, the environments match (the first of these that renders non-empty; Effective.EnvironmentConfigured reports that one did), the instance name, v1.DefaultEnvironment. An instance is therefore protected by the GitHub environment of its own name unless mapped elsewhere. A workspace: blue stack without instances therefore applies under blue; it sets environment: default to apply under default.
  • config.Resolve(root, path, stack, instance) (Effective, error) is pure and, through config.ResolveMatched(root, path, stack, instance, matchedVarFiles), which the scanner and the CLI call to add the matched var file, is the one place that merges and renders. path carries no suffix and an instance default means none. Effective carries Instance, EnvironmentConfigured, BackendConfig, VarFiles and Env (rendered, per mode; EnvFor(mode)). It does not need to know where an instance came from: the server calls it with the instance it splits off a key. config.EnvironmentFor returns the raw, unrendered environments value and never decides an environment. Environment variable names match [A-Za-z_][A-Za-z0-9_]* and may not start with STACKORDER_, GITHUB_, ACTIONS_ or RUNNER_, nor be PATH or HOME, compared without regard to case; default is refused as an instance name in any letter case, since GitHub environment names ignore case. A null list item under instances, a null env value, a null outputs and a null outputs.readers or item of it are errors.
  • Validation is strict (KnownFields): an unknown key is an error.

Scan and graph ​

  • stacks.exclude beats both stacks.discover and stacks.include.
  • The scanner globs from_var_files in each stack directory, computes the instance set and calls addStack once per instance.
  • The effective v1.Backend of an instance starts from the literals of the backend "s3" block and overlays the rendered backend_config entries in order: a file is parsed as HCL attributes and name=value sets one attribute; only bucket, key, region, dynamodb_table, use_lockfile and workspace_key_prefix are read, everything else is ignored. A missing file is a scan error. Inferred reads_state edges and the shared-state-object warning therefore work per instance, including two instances that render the same key.
  • WatchPaths holds the backend config files and the var files that the stack's own directory does not own, sorted and unique. TreeHash covers *.tfvars, *.tfvars.json and *.tfbackend files, and every backend config file, var file and from_var_files match the scan read, whatever its name. A changed path equal to a stack's watch path affects that stack with reason watch_path.
  • A rendered depends_on or ignore_inferred entry without a suffix that names a directory with instances resolves to the instance of the same name when the target has one, else to the target's only stack when it has exactly one; a depends_on entry that resolves to nothing is kept as written with a warning, as any unknown target is today, and an ignore_inferred entry that resolves to nothing suppresses nothing. A cross-repository entry is kept as written; it names its instance itself, with a template if it wants the same one. A cross-repository ignore_inferred entry, and a cross-repository depends_on that replaces an inferred edge, match the edge's target with owner/repo compared case-insensitively. An ignore_inferred entry qualified with this repository resolves as its unqualified key, and one that, cleaned as a path, names a stack of this repository (stacks/prod//vpc) keeps that reading. A cross-repository ignore_inferred entry that no stackorder_outputs block of the stack or of its local modules reads is a warning.
  • A data or ephemeral stackorder_outputs block, in .tf or .tf.json, infers a reads_state edge from each instance of its stack when its stack attribute is a literal string; override files replace a block's stack. A path or path:instance, or the same qualified with this repository, must equal a stack key exactly; an owner/repo//key of another repository, in canonical form with an owner/repo of GitHub's name characters, becomes an external stack. A non-literal stack warns to add depends_on; a key that names no stack, a malformed key (an empty repository part included) and a stack reading its own outputs are warnings without an edge. Blocks are also read in every local module the stack reaches through module sources, transitively, and infer the same edges from the stack; a module no stack reaches infers nothing, and the stack's own directory, reached again through a module, is not read twice. A block in a module whose stack refers only to var.<name> values takes them from the literal arguments of each module block that calls the module, one level up (stack = var.producer with producer = "stacks/vpc" in the call); a warning about the result points at that module block, once per call. Any other warning about a block in a module points at the block, once per block. Each names the module's directory (data.stackorder_outputs.vpc in module modules/lookup), and one about a non-literal, malformed or unknown stack says that only depends_on orders the stacks that call the module; like a direct block's warning, it stays after they add depends_on. A module directory that is also a stack directory does not repeat the warnings of that stack's own blocks. Registry and git module sources are not fetched, so a block inside one infers nothing and warns nothing, and the stack needs depends_on; terraform_remote_state is read only in the stack's own directory.
  • A listed var file that does not exist is a scan warning; the CLI errors at plan.
  • Affected reasons, in order: changed, watch_path (v1.ReasonWatchPath), module, reads_state, dependent, requested.

CLI ​

  • --stack path:instance, help text stack key: path or path:instance. With instances, declared in instances or derived from from_var_files, the suffix must name one and a bare path is an error listing them; with none, the suffix is a Terraform workspace (equal to workspace when set, ad hoc otherwise, and a valid instance name either way). check --stack resolves the key the same way, but only warns when the stack cannot be loaded for another reason, so a verdict can still be posted from a job without a checkout.
  • init passes Effective.BackendConfig, files resolved to absolute paths against the repository root, followed by STACKORDER_BACKEND_CONFIG, and -reconfigure whenever Effective.BackendConfig is not empty, so two instances can share one checkout; in that case it also resets the selected workspace to the instance's own (or default), since the checkout's .terraform may have been left on another instance's.
  • Results of every mode carry StackResult.Backend, read after init from terraform.tfstate in the tool's data directory, so -backend-config files and STACKORDER_BACKEND_CONFIG are already merged in, and, for an s3 backend with a bucket and a key, StackResult.State, whose key is the one the S3 backend of both tools writes (config.ResolvedStateObjectKey): the backend key as written in the default workspace, else path.Join of the workspace_key_prefix, the workspace the tool uses and the backend key. That workspace is found as the tool finds it: a non-empty TF_WORKSPACE in the tool's environment (EnvFor(mode) over the process environment), else the environment file in the data directory, which workspace selection writes and an earlier run may have left, else default. State is left out when workspace selection fails. The prefix is env: only when it is unset (null in terraform.tfstate); an empty string is no prefix. v1.Backend cannot tell the two apart, so keys computed from it (the scanner, the drift report, the UI) read an empty prefix as env:. The scanner and the drift report clean the key as State does, but the UI's stateObjectKey (ui/src/format.ts) joins the parts without cleaning them, so in a named workspace a key or prefix with dot segments or doubled slashes gives a state link to a different object than State.
  • plan, the re-plan inside apply and drift pass one -var-file per Effective.VarFiles; each must exist, and the error names it. apply <planfile> passes none. Plan-time values of non-ephemeral variables are frozen in the plan file; only ephemeral variables take the apply value of env on an apply from a saved plan.
  • The tool process and the hooks get EnvFor(mode) after the automation variables, and STACKORDER_STACK (the key), STACKORDER_STACK_PATH, STACKORDER_INSTANCE, STACKORDER_RUN_ID (the run id the CLI uses), STACKORDER_SERVER_URL (the --server value) and STACKORDER_REPOSITORY (owner/repo), each set even when empty, so an inherited value never disagrees with the CLI. Values of configured variables whose names match the secret pattern of envSecrets join the redaction set.

Server ​

  • Apply-time environment, allowed_teams and plan_output come from config.Resolve(defaultBranchRoot, path, defaultBranchStackConfig, instance), never from a field read off the raw file. The stack config is nil when the default branch has no stack file in that directory; the environment recorded on the plan row is never used. Dispatch grouping by environment, binding and deployment protection work as for any stack.
  • Runs whose keys share a directory are independent: separate locks, separate concurrency groups, separate artifacts.

Actions ​

  • aws-role-arn-map keys are prefix/, path:instance (an exact key) or :instance (that instance in any directory). Precedence: exact key, then :instance, then the longest prefix, then aws-role-arn. Plan and drift jobs keep the single plan role.
  • New input aws-role-session-name on both reusable workflows: a name or a JSON object with plan, apply and drift keys, passed as role-session-name after sanitising to [\w+=,.@-] and 64 characters.
  • The runner's credentials model for one OIDC bootstrap role and a provider assume_role chosen by an env variable is documented, not coded: the plan and apply roles are the two OIDC roles the workflows already take, the provider's role is TF_VAR_<name> with a plan and an apply value, and the variable is ephemeral so the apply value is used when applying a saved plan.

Actions repository interface ​

Composite actions resolve, plan, apply, drift take stack (except resolve), run-id, server-url (required everywhere), working-directory and github-token (default github.token, used only for the CLI's neutral fallback checks; not on apply). plan adds upload-artifact and retention-days (5); apply adds plan-run-id and artifact. resolve outputs matrix, waves, affected (JSON array of stack keys), count, run-id, unconfirmed, stackorder-version.

Reusable plan.yml inputs: server-url (required), aws-role-arn, aws-role-arn-map, aws-role-session-name, aws-region (us-east-1), tool, tool-version, stackorder-version, runner, max-parallel, working-directory, base-ref, stacks. Its resolve job needs pull-requests: read in addition to the design's four permissions; the caller's job must carry the permissions block because a called workflow cannot raise them. Fork pull requests are skipped with a step summary; the server posts the neutral check.

Reusable run.yml inputs: run-id, mode, wave, sha, stacks (required), server-url (required), aws-role-arn-map, aws-role-arn, aws-plan-role-arn, aws-role-session-name, aws-region, tool, tool-version, stackorder-version, runner, max-parallel, working-directory. Every job runs under environment: ${{ matrix.environment }} and in concurrency group stackorder-stack-<key> without cancel-in-progress. Role selection: apply uses the first match of aws-role-arn-map, whose keys are prefix/, path:instance or :instance, in the order exact key, :instance, longest prefix (a key with : is never a prefix), falling back to aws-role-arn; plan and drift use aws-plan-role-arn, falling back to aws-role-arn. plan.yml selects from the map the same way. aws-role-session-name is a name or a JSON object with plan, apply and drift keys (drift falls back to plan; plan.yml uses plan), sanitised to [\w+=,.@-] and 64 characters and passed as role-session-name; empty keeps GitHubActions. The key forms and the session name ship in stackorder/actions v1.0.0.

Job names carry the stack key last: run.yml names its jobs <mode> wave <n> <key> and plan.yml its plan jobs plan <key>. The server reads the key from these names, after the caller's <job> / prefix, or from GitHub's default matrix form <job> (<key>, …), to match workflow_job events and to choose among dispatches that share a title.

apply.max_parallel in stackorder.yaml caps the number of stacks the server puts in one dispatch; job level parallelism is the caller's max-parallel input, because dispatch inputs are fixed by the wrapper file.

Runner endpoints ​

Stack keys in paths are percent-encoded with url.PathEscape (slashes become %2F); Go's ServeMux matches the escaped path and r.PathValue returns the decoded key.

Method and pathBodyResponse
POST /v1/runsv1.CreateRunRequestv1.CreateRunResponse
POST /v1/runs/{id}/graphv1.GraphUploadRequestv1.ResolveResponse (200 with cycles set when resolution fails on a cycle)
PUT /v1/repos/{owner}/{repo}/default-graphv1.RecordGraphRequestv1.RecordGraphResponse
POST /v1/runs/{id}/stacks/{key}/resultv1.StackResultv1.RunStack
POST /v1/runs/{id}/stacks/{key}/checks/{name}v1.CheckVerdictv1.Check
GET /v1/runs/{id}v1.Run
GET /v1/repos/{owner}/{repo}/stacks/{key}/outputs?run=&reader=v1.StackOutputs of stack {key} of {owner}/{repo}, the producer, read as reader, a stack of run run (see Shared outputs); runner token or API key, which may omit run and reader, never a session; Cache-Control: no-store
POST /v1/unlockv1.UnlockRequest with repo and stack_keyv1.UnlockResponse

A manual run (POST /v1/runs with an API key, trigger: manual, mode: apply and stacks) makes the server take the locks before answering and release them when the results arrive; this is how stackorder apply --local works.

PUT /v1/repos/{owner}/{repo}/default-graph makes the graph a push-triggered resolve job scanned on the default branch the repository's default-branch graph, so a direct push to the default branch updates the stack lists, /v1/overview, removed stacks and drift scheduling at once instead of waiting for the next planned merge, which is the only other thing that moves repos.default_graph_id (runs.recordDefaultGraph). runs.RecordDefaultGraph accepts only a push token (event_name is push) whose repository and repository_id are the repository's and whose ref is refs/heads/<default branch>; the request graph's SHA must be the token's sha (it may be left empty to take it). It then reads the branch head from GitHub under the per-repository advisory lock default-graph:<repo id> (runs.atDefaultHead) and records the graph with store.SaveGraph and store.SetDefaultGraph only when the head still equals that SHA; otherwise the response has recorded: false and a reason, so an out-of-order, superseded or force push, whose commit is no longer the head, cannot replace a newer graph, and the last push to reach the head wins. The merge path (runs.recordDefaultGraph) shares this ordering through the same runs.atDefaultHead: it records the merged pull request's plan graph only while the merge commit is still the branch head, so a delayed or redelivered merge webhook that arrives after a newer push cannot revert the default branch to the pull request's older graph. In before_merge mode the merge releases the pull request's orchestration locks (runs.releaseMergeLocks) before this recording, so a failed head read, which makes runs.onMerged return an error and the queue retry the merge event, never keeps those locks held. SetDefaultGraph reconciles stacks.removed_at in both paths. A suspended installation's runner token is refused at authentication with 403 before the handler runs (internal/api/auth.go, like every runner endpoint), so a suspended installation records nothing. The server never scans a repository itself; the graph is always the one a resolve job uploads. No existing workflow posts here yet; a push trigger in stackorder/actions is a follow-up, and a repository that has not adopted it keeps the previous behaviour.

v1.StackResult.State is a v1.StateObject (bucket, key with the workspace prefix, region unless the backend takes it from the environment): the S3 object that held the stack's state after init and workspace selection, omitted when unknown. runs.RecordResult records it as the stack's stacks.state_object from an accepted apply or drift result, manual runs included and whatever the outcome, since a job that reports one got past init; never from a plan result or a graph upload. A refused or repeated result records nothing, and neither does a result of a run older than the run that recorded the current object. With a runs.StateReader, an object that StateReader.Allows refuses for the run's repository is not recorded either: store.ClearStackStateObject clears the stack's object instead, under the same run order, so reads answer state_location_unknown rather than serve an object the stack has left, and the location one repository reports is never another repository's state unless grants of different repositories overlap. An object without a bucket or key, one no grant of the repository covers, or a store error is logged at warn level without failing the result.

v1.StackResult.Resources is a v1.ResourceList (total, truncated, items of v1.StackResource: address, type, module, provider, tainted, deposed), the managed resources of the stack's persisted state, never an attribute value. The CLI reads it with tf.Runner.ShowStateJSONRaw (show -json -no-color with no file, through capture, so nothing is streamed or written) in apply jobs whenever the tool ran, failed applies included, after the post-apply hook, and in drift jobs after the plan whatever its outcome once init succeeded, each read under its own 5 minute timeout; plan jobs never set it. tf.Resources walks the modules without parsing addresses, tf.RedactResources redacts the four strings and merges addresses that collide, and finalize cuts the list with tf.FitResources to v1.MaxResultBytes minus the result without the list minus 64 KiB. Any failure warns and leaves it nil without changing the exit code; session.post posts once more without the list on a 413, or on a 400 invalid whose details.field starts with resources. runs.RecordResult records it for apply and drift runs, from an accepted result whatever the outcome, manual runs and both duplicate-result branches included, so a retry after a failed write records it; a refused result records nothing. cleanResources rebuilds each item from the whitelisted fields, dropping last_change, drops an item with an empty address or type, invalid UTF-8, a control or Unicode format character, an address or module over 2048 bytes or a type or provider over 512, sorts, merges duplicates by OR-ing their flags, clamps total between the item count and 10,000,000, and sets truncated when anything was dropped or left out. It warns once with counts and the first dropped item's index, field and reason, never an item's content. The write uses a detached 10 s context, and a failure is logged at warn level without failing the result. Unlike state_object, which follows run order, a list replaces the recorded one only when its observed_at is not older, computed on the database clock: now() for a confirmed apply result posted for the first time, whose job reads state after the apply and its hooks, right before posting; the stack row's run_stacks.finished_at, which that first accepted post stamped, for a confirmed apply result that arrives through either duplicate-result branch, falling back to the run's created_at when it is null; and the run's created_at for a drift or unconfirmed apply result, a lower bound on when its job read state. runs.resourceObservation picks the rule as a store.ResourceObservation. Run order would let a drift run created while an apply waited for approval, which may read state before the apply changes it, replace the apply's list; this rule rejects it, and any drift run created before an apply's list was recorded, even when it posts later. Dating a repeat by its first post means a late replay, such as an API key posting a finished manual run's result again, never replaces a list recorded by a drift run created or an apply posted since the stack finished, while a retry whose first list write failed still records unless such a list exists. The cost is discarding a drift list equal to the apply's.

v1.StackOutputs carries the root-module outputs of a stack: repo, stack, the state's serial, lineage and terraform_version, outputs mapping each name to a v1.StackOutput (value and type, the cty type as state writes it, both raw JSON, and sensitive), an empty object rather than null when there are none, and withheld, the names of the sensitive outputs the producer withholds, omitted when empty. api/v1/testdata/stack_outputs.json and stack_outputs_empty.json pin its wire format.

Authentication: Authorization: Bearer <GitHub OIDC token> for runners, Authorization: Bearer sk_<key> for automation, cookie stackorder_session for humans. An automation key may be scoped read-only, limited to one installation and given an expiry (see Human and automation endpoints). Every runner token is accepted once (its jti is recorded), so the CLI requests a fresh token for every call. Errors are v1.Error with codes unauthorized (401), forbidden (403), not_found (404), invalid (400), conflict (409), refused (409, apply gate; details.failures), locked (423, details.conflicts), superseded (409), unprocessable (422), internal (500), unavailable (503, JWKS, database or a state object unreachable). Reading a stack's outputs refuses with the generic codes above and the v1.ErrCode* codes outputs_disabled (404), outputs_not_shared (403), state_location_unknown (409), state_bucket_not_allowed (403), state_not_found (404), state_access_denied (403), and state_encrypted, state_unsupported, state_too_large and state_unreadable (422); a state read that fails for now, such as an S3 throttle or 5xx, is 503 unavailable. A principal.CodedError answers with its own code, message and details and the status of its Kind: ErrNotFound 404, ErrForbidden 403, ErrConflict 409, ErrUnprocessable 422, ErrUnavailable 503, any other 500. The CLI's internal/client retries network errors, timeouts, 429 (from a rate limiter or reverse proxy in front of the server) and 5xx three times, honours a Retry-After header in either form, capped at a minute, and treats a 429 or 5xx that persists through the retries as unreachable. The CLI exits 3 on forbidden, conflict, refused, locked and superseded, and on its own fail-closed decisions (an unreachable server during apply). invalid carries details.field and details.reason when a field is at fault, a JSON type error in a body included, whose field is its path without array indexes, such as resources.items.tainted, the same whichever Go release built the server. A body over its route's limit (8 MB for graph uploads, 1 MB, 1 << 20 bytes, for every other POST under /v1 and /auth, results included through v1.MaxResultBytes, which has the same value) is 413 with code invalid; POST /webhooks/github has no API limit, internal/webhook bounds it.

Human and automation endpoints ​

Sessions or API keys. Lists take ?limit=&cursor= and return v1.Page[T] with next_cursor.

Method and pathResponse
GET /v1/mev1.Whoami; for an API key login is apikey:<name> and admin is true
GET /v1/overviewv1.Overview
GET /v1/reposv1.Page[v1.RepoSummary]
GET /v1/repos/{owner}/{repo}/graph?ref=&run=v1.GraphView; ref is a full commit SHA, a unique prefix of at least 7 characters of a recorded graph's SHA (ambiguous: 400 invalid) or default (the default-branch graph), never a branch name; without ref the repo's default-branch graph when known, else the latest
GET /v1/repos/{owner}/{repo}/runs?status=&pr=&mode=v1.Page[v1.Run], newest first
GET /v1/repos/{owner}/{repo}/stacksv1.Page[v1.StackDetail]
GET /v1/stacksv1.Page[v1.StackDetail], the live stacks of every visible repo, ordered by repo full name, then key
GET /v1/stacks/{id}v1.StackDetail
GET /v1/stacks/{id}/runsv1.Page[v1.RunStackRef], newest first
GET /v1/stacks/{id}/resourcesv1.StackResources: the recorded list with each item's last_change from store.StackResourceChanges; 200 with items: [], total 0 and no run fields when nothing is recorded; a removed stack keeps its last list; 404 through visibleStack; not paged, since a result body bounds the list
GET /v1/modules?q=v1.Page[v1.ModuleDetail]
GET /v1/modules/{id}v1.ModuleDetail
GET /v1/runs/{id}v1.Run
GET /v1/runs/{id}/stacks/{key}/plantext/plain: the run stack's full plan text, read from the artifact bucket at store.PlanTextArtifactKey (runs/<run id>/<slug>/plan.txt); 404 not_found when the row has no plan_url, the server has no artifact bucket or the object is gone
POST /v1/stacks/{id}/unlockv1.UnlockResponse
POST /v1/runs/{id}/rerunv1.CreateRunResponse
GET /v1/auditv1.Page[v1.AuditEntry]
GET /v1/keysv1.APIKeyList; unscoped administrator API key only
POST /v1/keysv1.CreatedAPIKey (201), the secret shown once; v1.CreateAPIKeyRequest (optional read_only, installation_id, expires_at); unscoped administrator API key only
DELETE /v1/keys/{id}v1.APIKey with revoked_at set; unscoped administrator API key only
GET /auth/login?next=, GET /auth/callbackredirects
POST /auth/logout204
GET /setup, GET /setup/callback, GET /setup/installedHTML
POST /webhooks/github202
GET /healthz, GET /readyz, GET /metricssee below

A session user sees only repos whose installation account is one of the user's organisations (or the user's own login). An unscoped API key sees every installation; an installation-scoped key (api_keys.installation_id set) sees only that installation and nothing of others — the server resolves the installation to its account in apiKeyIdentity and the key then reuses the session visibility machinery through identity.accounts/identity.sees, so repository and stack lists, overview counts, runs, plan text, shared outputs, resources and the audit log are all filtered, and anything of another installation answers 404 not_found exactly as an absent object does, with no 404-versus-403 existence oracle. A key whose installation is no longer installed sees nothing (the scope never collapses to the every-account filter). A session records the organisations at sign-in, so a membership change takes effect at the next sign-in. For a person, GET /v1/audit keeps the entries they made and those whose details.repo, or target (owner/repo… or kind:owner/repo…), names a repository they see, so a page may hold fewer than limit items and still carry next_cursor; GET /v1/modules sets total only for unscoped API keys.

A key is scoped in three independent ways (migrations/0016_api_key_scopes, columns read_only, installation_id, expires_at, all permissive by default so rows from before the migration stay unscoped administrators). A read-only key (read_only) is refused every mutating route centrally in with: an API key whose request method is not GET/HEAD is 403 forbidden, which covers every route in the authRoutes table, including one added later, without a per-handler check. An expires_at in the past makes store.VerifyAPIKey refuse the key exactly as a revoked one, so an expired key is 401 unauthorized; expiry is re-checked on every request.

State-changing human endpoints require the session cookie plus a same-origin Origin or Sec-Fetch-Site check. The /v1/keys endpoints are the one administrator-only surface: they carry the adminKey auth mode, which with enforces after authentication, so a session is 403 forbidden, a runner token 401, and only an unscoped API key — identity.isAdmin, read-write, installation-wide and non-expiring — manages keys. A read-only, installation-scoped or expiring key is refused, so a restricted key can neither list, create nor revoke keys, and therefore can never obtain a broader key, and a time-boxed key can never mint a permanent one. isAdmin matches store.APIKey.Admin and is the single chokepoint the scope narrows. Creating and revoking a key is audited as api_key_created and api_key_revoked with the key name as the target and the scope in details; the store writes the mutation and its audit entry in one transaction, so a key never exists without its creation entry and the audit record is only ever written by the call that takes effect, which means a repeated revoke of an already-revoked key records nothing. The secret is returned once at creation and never stored in the clear. The same operations, with the same scope flags, are available offline through stackorder-server api-key. Unknown /v1/* and /auth/* paths return a JSON 404, not the UI. Everything else under / serves the embedded UI with SPA fallback to index.html. GET /metrics is open unless STACKORDER_METRICS_TOKEN is set, in which case it requires Authorization: Bearer <token>. GET /readyz returns 200 when the database answers, in setup mode too. GET /setup shows the App manifest form, in setup mode and for ?force=1 with STACKORDER_ALLOW_RESETUP, only to a browser holding the setup token: ?token=<token>, compared in constant time, sets the signed stackorder_setup_auth cookie (path /setup, one hour, bound to the token) and answers 303 to the same URL without the token; without the token or the cookie the page answers 403. GET /setup/callback converts a code only with the signed state cookie the form page set, and finishes the conversion even if the browser disconnects. While a conversion runs, /setup and other callbacks answer 409 and keep their cookies, so a reload after a failed conversion retries the same code. Once a callback has created an App the token and its cookie stop working on that instance until it restarts; its /setup then says an App exists and its callback answers 409. The flag is in memory, so other instances sharing the token keep accepting it until they restart, which is why one instance during setup is the recommendation. GET /setup/installed needs no token: it echoes only the installation id GitHub passes.

Run state machine ​

Run status: pending -> planning -> planned -> applying -> applied, with failed reachable from planning and applying, unconfirmed reachable from pending and planning, and superseded reachable from any non terminal state when a new head SHA arrives for the same PR.

Stack status within a run: pending -> planning -> planned -> applying -> applied; failed from planning and applying; blocked when a transitive depends_on or reads_state predecessor failed (v1.RunStack.BlockedBy names them); noop when a propagated stack's plan is empty at apply time; unconfirmed when the CLI reported without a confirmed server round trip; unknown when the job vanished; skipped when the requester named a subset that excludes the stack.

Roll-up: a run is planned when every stack is planned, noop or skipped; applied when every stack is applied, noop or skipped; failed when any stack is failed, blocked or unknown and no stack is still running.

When a PR merges into the default branch, its latest graph becomes the repository's default-branch graph (repos.default_graph_id), which the graph endpoint and module consumer queries prefer over the newest PR graph. The default branch is the one the event's repository names, which also updates repos.default_branch; a merge into any other branch counts as a close without merge.

Per-stack policy (allowed_teams, plan_output) comes from the default branch: the default-branch graph when known, else the .stackorder.yaml files read through the Contents API at the default-branch head. A default-branch plan_output: summary always wins over the PR's copy. A pull request is resolved under the stackorder.yaml it uploads; when that differs from repos.config after defaults, it is resolved under both and the affected sets are united, stacks of the default-branch graph (else the latest graph) that only repos.config discovers and finds affected are added to the commit's graph, and the run warns with the keys that differ. Such a stack whose directory the pull request deletes (a changed path under it, and the directory gone at the head per the Contents API) is left out of the graph and the run warns that it is recorded as removed. workflow_job events of the pull-request plan workflow, matched through runs.workflow_run_id, move stacks to planning. Named check verdicts may not use the reserved names resolve, plan or apply. A pull request has at most one apply in flight. An apply of a PR that affects nothing is answered with a comment, while a before_merge PR whose head affects nothing still gets a successful stackorder/apply check so branch protection can pass.

Apply gate, in order ​

  1. A requester is required (an empty requester fails; API keys are trusted) and may apply every affected stack: allowed_teams membership (active, nested teams count, cached 60 s) or push permission when the list is empty.
  2. The PR is open and not merged, mergeable is not false and mergeable_state is not dirty (blocked is not a refusal, because stackorder/apply is itself a required check); approvals on the head SHA from users with push permission (a later CHANGES_REQUESTED or a dismissal cancels one) >= require_approvals; four_eyes and require_codeowner_review (owner reviews also only from users with push permission) when set.
  3. Every affected stack has a planned row for the current head SHA, with a plan artifact when apply.from_plan is true.
  4. Every named check on those stacks is pass (warn passes, fail refuses).
  5. No affected stack is locked by another PR, and no other apply of this PR is in flight.

All failures are collected and reported together in one PR comment naming the failing layer and the exact reason. Policy for the gate is read from the default branch stackorder.yaml (repos.config), never from the PR's copy. In on_merge repositories stackorder apply comments are refused; the merge evaluates layers 1 (with the merger), 3, 4 and 5, applies the merge commit with the head commit's plans, and releases the locks on completion. Comment commands are accepted only from users with push permission, then rate limited to 10 per PR per minute (counted from audit rows); a command runs at most once, and one that fails part way is answered with a comment.

GitHub client ​

internal/gh talks to the REST API over net/http. It retries up to five attempts with exponential backoff and jitter, honouring Retry-After and X-RateLimit-Reset, but only for requests that are safe to repeat: reads and the PUT/DELETE endpoints, the PATCH of a check run, comment or issue (idempotent in effect, since the body sets fixed fields on a known object and the server sends no annotations, which GitHub would append), the POST of a reaction (GitHub answers 200 with the existing reaction), the installation-token exchange (minting another token is harmless) and the deployment-protection-rule review (a repeat is refused with 422, which the caller already treats as answered). A non-idempotent create — a new check run, comment, issue, or a workflow dispatch — is retried only while the request provably has not reached GitHub (a connection error before the request body was written, detected with httptrace) or when GitHub reports it did not act (429, or a primary or secondary rate limit). After any other failure the error is returned, marked with ErrUnknownOutcome so a caller can tell that GitHub may have applied the create. Only the App-manifest conversion is never retried at all, because a code converts once.

Callers then retry whole operations, so the change removes the duplicate object rather than moving it into the worker; where a surplus cannot be prevented, as with a resent dispatch's workflow run, binding makes it harmless (see Waves and dispatch). When a check-run create's outcome is unknown, the renderer looks for a check run carrying the run's external_id and adopts it before giving up. The run comment and sticky comment are found again by their marker, a drift issue by its title and a fork notice by its check-run name before a create; the stale-lock reminder records its dedup audit row, and a resolved drift issue is closed, before the comment is posted, so a lost response to either cannot post it twice. A workflow dispatch left unsent is resent once by Reconcile, and run binding refuses a surplus workflow run (see Waves and dispatch).

Waves and dispatch ​

Waves are the longest-path layering of the affected subgraph over depends_on and reads_state edges. The server dispatches stackorder-run.yml once per (wave, environment) with inputs run_id, mode, wave, sha and stacks (JSON array of v1.MatrixEntry carrying plan_run_id and artifact for applies), at most apply.max_parallel stacks per dispatch. Wave n+1 is dispatched when every stack of wave n is terminal and none failed. Locks are taken on all affected stacks before wave 0 is dispatched and released on merge (before_merge) or run completion (on_merge and manual runs). A stackorder plan comment dispatches mode: plan for the named stacks under environment default; the scheduler dispatches mode: drift per stack under default with sha set to the default branch head.

Dispatches are unique per (run, wave, environment, mode, chunk), where chunks split one (wave, environment) by apply.max_parallel; sent_at marks a dispatch GitHub accepted, and Reconcile resends unsent ones after 60 s. An apply run (manual runs excepted) with no dispatch row two minutes after it was created, still pending or applying with stacks of its current wave planned, has that wave dispatched by Reconcile while it is the pull request's newest apply, a comment apply's pull request is still open at the run's commit and every planned stack keeps the head's plan; otherwise it is failed, its locks are released (unless an earlier apply of the pull request changed what is deployed) and the pull request gets a comment. Each dispatch row binds at most one Actions workflow_run_id: a workflow_run event's display_title binds only when it names a single dispatch of that (run, wave, mode); otherwise the stacks named by the run's jobs choose the dispatch, or the environment of a deployment_protection_rule does, and Reconcile never binds by position. The first OIDC call of a job also binds; a call that names no stack, or a stack no dispatch carries, where the environment has several dispatches in the wave, binds by the stacks the workflow run's jobs name. A workflow run whose dispatch is already bound binds nothing and its jobs are refused. The gh client does not retry the dispatch POST once it may have reached GitHub (see GitHub client), so a processed 5xx no longer dispatches twice; a dispatch left unsent by an error is resent by Reconcile, and this binding makes that duplicate harmless when GitHub had in fact accepted the first. Cross-repo plan runs use trigger push and are deduplicated per upstream run and repository.

Shared outputs ​

runs.Service.StackOutputs(principal, repo, key, run, reader) serves the root-module outputs of a stack as v1.StackOutputs, behind GET /v1/repos/{owner}/{repo}/stacks/{key}/outputs?run=&reader=. It refuses everything with outputs_disabled (principal.ErrNotFound) unless the service has a runs.StateReader (runs.WithStateReader), which the server passes only when STACKORDER_STATE_BUCKETS is set. In order:

  1. Binding. The repository, key, run id and reader are valid UTF-8 without control characters, else principal.InvalidError before anything is looked up. A runner token names its run and the reader, a stack of that run. The token carries the run's repository: a run of another repository id is refused with the message of a run that does not exist and the reason is logged, while a token that names the run's repository id under another name is refused with both names; the run is not superseded (superseded), not terminal, and for a pull_request token not planned either (conflict); the reader's row in the run has not reported (pending, planning or unknown in a plan or drift run; pending, planned, applying or unknown in an apply run) and, for a workflow_dispatch token, was dispatched; then the job binds to that row as for its results (authorizeJob), except that a read binds a workflow run to the reader's dispatch on first contact only when the workflow run's jobs name a stack of that dispatch. An API key is an admin: a run id and a reader, when given, are only checked for form. Other principals are refused.
  2. Producer. The repository, matched without case, exists, is not suspended and, for a runner token, belongs to the reader's installation; the key is local and has a stack row.
  3. Default branch and policy. Everything here is read through the Contents API with the producer's installation at repos.config_sha (cached 24 h per repository, commit and directory), or at the default branch, cached 60 s, when no commit is recorded. The producer must be a stack the default branch still has at that commit: repos.config discovers the key's directory (stacks.discover, include, exclude and modules.paths, as the scanner applies them), the directory exists (its .stackorder.yaml shows it, else a listing of the directory itself, gh.Client.ListDir), and the key's instance is one config.InstanceNames gives for that .stackorder.yaml and, when the stack declares no instances, for the stacks.instances.from_var_files matches config.MatchVarFilesFS finds in the directory's listings at that commit, each listing cached like the stack file. A merged pull request's graph (repos.default_graph_id), which lags the default branch and which pull requests upload, plays no part. A listing GitHub fails is an error, never a refusal. outputs.readers and outputs.sensitive then resolve through config.Resolve from repos.config and that .stackorder.yaml. Directory and instance come from the key, never from stacks.path, stacks.config, a graph or GraphUploadRequest.config, which pull requests write. A configuration that does not parse or render shares nothing. config.ReaderAllowed must accept the reader; API keys skip it.
  4. Location. The stack's stacks.state_object, else state_location_unknown (principal.ErrConflict), whose message says whether no job reported an object or the last report was cleared as not granted (stacks.state_object_at set); stacks.backend is never read.
  5. Read. StateReader.Read for the producer's repository, which refuses a location no grant of that repository, or of *, covers with stateread.ErrNotAllowed before any request to S3; that is state_bucket_not_allowed (principal.ErrForbidden), audited as a refusal and logged at warn level with the repository, stack, bucket and key. Its other sentinels map to state_access_denied (principal.ErrForbidden), state_not_found (principal.ErrNotFound), and state_encrypted, state_unsupported, state_too_large and state_unreadable (principal.ErrUnprocessable); any other error is principal.ErrUnavailable, with a generic message, and is logged.
  6. Response. Sensitive outputs are left out and named, sorted, in withheld unless outputs.sensitive is share; outputs is never nil.

Refusals carry their code in a principal.CodedError, whose Kind is the sentinel the API maps to a status. A reader of the producer's repository that the policy does not list gets outputs_not_shared (principal.ErrForbidden), naming the files to edit, as does any reader of that repository when the policy does not resolve; a producer the default branch no longer has is not_found for every caller, inside the repository too, and across repositories every refusal of steps 2 and 3 is not_found with one message for every cause. Reads are audited as outputs_read with the number of outputs served, sensitive outputs served and outputs withheld, and refusals of steps 1 to 4, and of step 5 for a location no grant covers, as outputs_refused with the code as reason; details.repo is the producer when the caller may see it, else the caller's repository, so the producer's organisation sees reads of its stacks. Names the caller sent (repository, stack, reader, run id) have control characters and invalid UTF-8 replaced by U+FFFD and are then cut to 256 bytes in the audit row and its target. Outputs are served only once their outputs_read row is written: a failed write is logged at error level and the read answers principal.ErrUnavailable; a refusal whose row cannot be written is logged the same way and still refused. Nothing read from a state object is stored, cached or logged.

OIDC binding ​

GitHub's sha claim is the commit that triggered the workflow: the merge commit of refs/pull/<n>/merge for pull_request events and the default branch head for workflow_dispatch. It is therefore never compared with a run's SHA. The server verifies the runner token signature against the GitHub JWKS (cached 1 h, refreshed on unknown kid), then checks: aud equals STACKORDER_OIDC_AUDIENCE; iss is https://token.actions.githubusercontent.com (or GITHUB_OIDC_ISSUER); repository and repository_id belong to a known installation; iat is within 10 minutes; jti has not been seen; job_workflow_ref matches STACKORDER_REQUIRED_WORKFLOW_REF when set. Then per run kind:

  • Plan runs: event_name is pull_request, ref is refs/pull/<run.pr_number>/merge, and the PR's current head SHA fetched from GitHub equals the run's SHA (the CLI registers plan runs with the head SHA from the event payload). pull_request tokens reach only plan runs registered by a pull-request resolve job; every server-dispatched run, plan runs included, needs the workflow_dispatch binding. Results for a terminal run are refused, and for a pull_request token a plan run that is planned is final too: its results, check verdicts and GET /v1/runs/{id} answer 409 conflict (superseded for a superseded run), except a repeat of the recorded result or verdict, which gets the stored row. Superseding compares against the PR head fetched from GitHub, never against an event's SHA, and default-branch pushes load stackorder.yaml at the branch head.
  • Default-branch graph records: event_name is push, ref is refs/heads/<default branch>, and the request graph's SHA is the token's sha. The server then reads the branch head from GitHub and records the graph only when the head still equals that SHA, so a PR token (refrefs/pull/<n>/merge), a fork (which cannot mint a token for the base repository), or a push whose commit is no longer the head records nothing.
  • Dispatched runs: event_name is workflow_dispatch, ref is refs/heads/<default branch>, run_id is the workflow run of the dispatch that carries the stack (binding it on first contact; in an apply, a stack no dispatch carries takes a dispatch of its wave and a skipped stack takes none, so a job never acts for a stack of another wave or one the apply skips), run_attempt is recorded, and for apply the environment claim equals the stack's environment; for plan and drift dispatches it equals default.
  • Stack outputs reads bind the token to the reader's row of the named run the same way (see Shared outputs). A pull_request token reaches any unreported stack of its pull request's unfinished plan run, so the reader is proven only to be a stack of that run; a workflow_dispatch token reaches only a dispatched, unreported stack of the dispatch its workflow run is bound to, which is exact when apply.max_parallel is 1. A read never makes that binding on the reader's word alone: a workflow run no dispatch holds yet is bound by a read only when its jobs name a stack of the reader's dispatch, so a job whose workflow_run webhook was lost or late cannot read as a stack of another chunk and take that chunk's dispatch. A noop or blocked stack of the same wave and environment, which the binding alone accepts for any chunk, is refused by its status, and an undispatched one because no dispatch carries it.

actor is recorded as requested_by. pull_request_target plans are not supported; fork pull requests get a neutral check from the server.

Metrics ​

All names carry the stackorder_ prefix: runs_total{status,trigger,mode} (each status a run enters, so a transition count, not a run count), runs_created_total{trigger,mode} (one per run row created), stack_finished_total{mode,status}, stack_duration_seconds{mode}, dispatches_total{mode,result}, drifted_stacks, locks_held, commands_total{verb,accepted}, webhook_received_total{event}, webhook_duplicates_total, webhook_lag_seconds (received to a worker starting on the event), events_processed_total{kind,result}, events_dead_total{kind}, jobs_processed_total{kind,result}, queue_depth{queue} (refreshed once a minute), artifact_operations_total{operation,result} (reads and writes against the artifact bucket), github_requests_total{route,status}, github_request_duration_seconds{route} (routes as METHOD /template), github_rate_limit_remaining{installation} (the installation id, or app for the App JWT), scheduler_leader, http_requests_total{route,method,status}, http_request_duration_seconds, build_info{version,commit}. Label values outside a known set are counted as other, except installation, which is recorded verbatim and so is unbounded.

Server configuration ​

VariablePurpose
STACKORDER_BASE_URLPublic URL, used for OAuth callbacks, OIDC audience default and links
STACKORDER_LISTENListen address, default :8080
DATABASE_URLPostgres DSN; sslmode=require is accepted
GITHUB_APP_ID, GITHUB_APP_PRIVATE_KEY, GITHUB_WEBHOOK_SECRETApp identity; all three absent puts the server in setup mode
GITHUB_OAUTH_CLIENT_ID, GITHUB_OAUTH_CLIENT_SECRETHuman sign-in
GITHUB_API_URL, GITHUB_WEB_URLDefault https://api.github.com and https://github.com; GHES sets its own
GITHUB_ENTERPRISEtrue or false forces whether the server is treated as talking to GitHub Enterprise Server; unset infers it from GITHUB_WEB_URL, every root other than https://github.com counting as Enterprise Server. It decides what internal/gh.IsEnterprise returns, which drives the App manifest the setup flow offers
GITHUB_OIDC_ISSUER, GITHUB_OIDC_JWKS_URLOverrides for GHES and tests
STACKORDER_OIDC_AUDIENCEDefault STACKORDER_BASE_URL
STACKORDER_REQUIRED_WORKFLOW_REFOptional glob pinning job_workflow_ref, validated at start-up
STACKORDER_ARTIFACT_BUCKET, STACKORDER_ARTIFACT_PREFIXOptional S3 bucket (and key prefix) for full plan text
STACKORDER_STATE_BUCKETSComma-separated owner/repo=bucket or owner/repo=bucket/prefix entries, * in place of owner/repo for every repository: the state objects the server may read to serve the stack outputs of each repository; unset turns shared outputs off
STACKORDER_STATE_ROLE_ARNOptional IAM role the server assumes to read those state objects; requires STACKORDER_STATE_BUCKETS
STACKORDER_STATE_EXTERNAL_IDOptional sts:ExternalId sent when assuming STACKORDER_STATE_ROLE_ARN; requires it
STACKORDER_SESSION_KEY32 byte hex key for cookie signing; generated and logged as a warning when absent
STACKORDER_SESSION_TTLSession cookie and server-side lifetime, default 168h, between 5m and 8760h
STACKORDER_METRICS_TOKENWhen set, /metrics requires this bearer token
STACKORDER_HSTS_MAX_AGE, STACKORDER_HSTS_INCLUDE_SUBDOMAINS, STACKORDER_HSTS_PRELOADOptional Strict-Transport-Security; max-age unset sends no header, sent only when the base URL is https, and set with an http base URL is an error; preload needs includeSubDomains and a max-age of at least 8760h
STACKORDER_ALLOW_RESETUPtrue lets a server with App credentials create another App through /setup?force=1; default false, and then /setup?force=1 and /setup/callback answer 404 outside setup mode
STACKORDER_SETUP_TOKENOne-time bootstrap token /setup requires, at least 32 URL-safe characters; when unset and /setup can create an App, 32 random bytes, base64url encoded, generated at start
STACKORDER_PLAN_TEXT_RETENTION, STACKORDER_EVENT_RETENTION, STACKORDER_DRIFT_RETENTIONDurations, defaults 720h, 168h, 2160h
STACKORDER_SCHEDULER_CATCH_UPHow far a new scheduler leader looks back for missed schedules, default 1h, at most 168h; only the latest fire of each schedule in the window is enqueued
STACKORDER_WORKERSWorker goroutines, default 4; the pgx pool gets this many connections plus 8 unless DATABASE_URL sets pool_max_conns
STACKORDER_LOG_LEVEL, STACKORDER_LOG_FORMATinfo / json by default
OTEL_EXPORTER_OTLP_ENDPOINTEnables tracing; spans carry stackorder.run_id

The server refuses to start without DATABASE_URL and STACKORDER_BASE_URL; retentions must be positive durations, as must STACKORDER_SCHEDULER_CATCH_UP, which is also at most 168h; STACKORDER_SESSION_TTL must be a duration between 5m and 8760h; STACKORDER_HSTS_MAX_AGE set with an http base URL, or the HSTS sub-settings set without it, are errors; STACKORDER_ARTIFACT_PREFIX without a bucket is an error; a partial set of the three GITHUB_APP_* variables is an error. Without all three it starts in setup mode, runs migrations, and serves only /setup*, /healthz and /readyz. Whenever /setup can create an App, in setup mode or with STACKORDER_ALLOW_RESETUP, the server logs setup_url at warn level once at start-up: the setup URL with the generated token, or with a <STACKORDER_SETUP_TOKEN> placeholder when the operator passed the token. The default OIDC audience is the base URL with trailing slashes removed, and the /setup page tells operators to use that exact value as the Actions server-url. A generated session key is warned about, never logged. With an artifact bucket or state buckets, AWS_ENDPOINT_URL_S3 switches the artifact store and the state reader to path-style addressing at that endpoint (LocalStack); the state reader's STS requests for STACKORDER_STATE_ROLE_ARN keep the SDK's own endpoint resolution (AWS_ENDPOINT_URL_STS), so an S3 endpoint does not redirect the role assumption. Artifact URLs are s3://bucket/key. Each entry of STACKORDER_STATE_BUCKETS is a stateread.Grant: a repository, owner/repo compared without case or * alone for every repository, then = and a valid S3 bucket name, optionally followed by / and a key prefix without a . or .. segment; a bare bucket[/prefix] entry is an error that shows the accepted forms. A prefix covers a key that equals it or continues it past a / boundary, so stacks/prod does not cover stacks/production/x. STACKORDER_STATE_ROLE_ARN must be the ARN of one IAM role, and setting it without STACKORDER_STATE_BUCKETS is an error. STACKORDER_STATE_EXTERNAL_ID must be 2 to 1224 of the characters AWS allows in an STS external ID and is sent as the ExternalId of the role assumption; setting it without STACKORDER_STATE_ROLE_ARN is an error, and the server never logs its value. With state buckets the server builds a stateread.Reader for the listed grants, assuming STACKORDER_STATE_ROLE_ARN when set, with STACKORDER_STATE_EXTERNAL_ID as its external ID, passes it to the run service and logs once at start-up, at info level, that shared outputs are on, with the number of grants, the entries as repository=bucket/prefix and the role, then, at warn level, each pair of grants of different repositories that stateread.Grant.Overlaps finds, since the stacks of either repository can then be pointed at the objects both cover. stackorder-server healthcheck GETs /healthz on the listen port (an unspecified host maps to 127.0.0.1) and exits 0 on 200, for container health checks in images without a shell; the Dockerfile declares it as HEALTHCHECK. stackorder-server migrate applies the embedded migrations to DATABASE_URL and exits, under the migration lock, needing neither STACKORDER_BASE_URL nor the GitHub App settings; --to N stops at a version, --status reports the applied and target version and exits 0 up to date, 2 pending, 1 dirty or ahead of the embedded latest, and --down --to N reverts the schema to version N for an operator rolling back to an older release, printing the plan and the data each step drops unless --yes is given. --force N records version N and clears the dirty flag on a dirty database, for recovering a migration that failed. At start-up, and daily at 04:00 UTC, the server syncs installations and their repositories from the App API (adding and refreshing, reading stackorder.yaml only for repositories not seen before) so installations made in setup mode or lost webhooks are learned; when the installation listing and the repository listing of every unsuspended installation succeed, it also forgets a stored installation the App no longer lists, but only after GET /app/installations/{id} confirms it with a 404 (a 200 or any other answer keeps it), and stored repositories of an unsuspended installation that no listing names. Omission is measured against the installations and repositories stored before the sync, so repositories the same sync newly discovers do not dilute it. A listing that omits a majority of the stored installations is treated as degraded and keeps every one of them, though their repositories are still reconciled from each listed installation's own listing; a listing that omits a majority of a listed installation's repositories keeps that installation's repositories, and because a repository has no per-repository confirmation, a listing that omits all of them keeps them as well, so an empty or short 200 cannot delete a single-repository installation's only repository. When any listing fails it forgets nothing; InstallationRepos fails a repository listing whose pages hold fewer repositories than the total_count GitHub reports. Traces carry stackorder.run_id, stackorder.event, stackorder.delivery, stackorder.job and stackorder.job_id; /healthz, /readyz and /metrics are not traced. A queued row keeps the W3C trace context of the request or job that enqueued it in its trace_context jsonb column (migration 0015, nil when tracing is off); the webhook handler and worker.Pool.Enqueue capture it through Server.injectTrace, and Server.extractTrace makes each worker span a child of that remote parent, so a delivery or job and its handler share one trace id, a retry is a new child, and rows from before the upgrade stay root spans. server.NewLogger wraps the slog handler so records logged with a span in context also carry trace_id and span_id. The pgx pool has STACKORDER_WORKERS + 8 connections (server.PoolConnsBeyondWorkers) unless the DSN's pool_max_conns sizes it; the scheduler's leader lock takes one further connection outside the pool (store.TryAdvisoryLock opens its own pgx.Conn), so a running instance opens at most pool_max_conns + 1. The leader holds that connection; a non-leader opens and closes one every tick (30 s) to test the lock. The start-up migration lock (store.Migrate) takes a dedicated connection the same way, before the scheduler runs, so at most one connection beyond the pool is open at a time.

Store notes ​

/migrations is a Go package exposing migrations.FS; store.LatestMigration reports the highest embedded version without a database. store.Migrate takes store.MigrationLockKey; store.MigrateUp takes the same lock, reads the applied version under it and migrates up to a given version, reporting whether it applied anything and refusing a dirty schema with a store.SchemaDirtyError, or one already above the target with a store.SchemaAheadError, instead of reverting anything; store.Migrate (the start-up path) refuses a dirty schema the same way. store.ForceVersion reads the dirty flag and records a version, clearing the flag, in one locked operation without running a migration (version 0 means no migration applied), returning the version the schema was dirty at, for recovering a dirty schema after the operator has brought it to a known version; it refuses a schema that is not dirty with a store.SchemaNotDirtyError so a delayed or concurrent recovery cannot rewind an already-clean database. A migration file runs as one implicit transaction under the pgx/v5 driver, so a failed migration rolls back and leaves only the dirty flag at the previous version. store.SchemaVersion reports the applied version and whether it is dirty, reading schema_migrations under the lock without creating it. store.MigrateDownTo takes the same lock and reverts to a given version (version 0, like store.MigrateDown, reverts everything), refusing a dirty schema and refusing to migrate up with a store.SchemaBehindError; store.Migrations lists every embedded migration's version and name. The stackorder-server migrate subcommand applies them up to the latest or a --to version, reverts with --down --to N, reports --status, or recovers a dirty database with --force, needing only DATABASE_URL. The scheduler takes store.SchedulerLockKey through store.TryAdvisoryLock, which returns (*AdvisoryLock, acquired, err) with Held(ctx) and Release(). Dispatches are unique per (run, wave, environment, mode) and a workflow run id belongs to at most one dispatch. Sessions store sha256(token); API keys are sk_ plus 43 base64url characters with the sha256 stored and a 10 character display prefix. Edges store from_key and to_key text so they can point at external stacks; backends are a backend jsonb column. Job dedupe_key stays taken until the job is pruned, so recurring jobs put a time slot in the key. store.ReleaseLockOfPR checks the holder and deletes in one statement. Queue: worker ids are host:pid:n; a failed row retries after 1 m, 5 m, 30 m and 2 h and becomes a dead letter after 5 attempts; shutdown hands unstarted claims back through store.ReleaseClaims and cancels handlers after a 30 s drain; the stale claim age (10 m) exceeds the handler timeout (5 m); store.QueueDepth feeds the gauges. Scheduler: cron runs in UTC unless the expression carries a CRON_TZ= prefix; a new leader catches up over STACKORDER_SCHEDULER_CATCH_UP (1h by default, at most 168h), keeping the latest fire per window; dedupe keys are reconcile:<minute>, prune:<hour>, stale_locks:<day>, schedule_drift:<repo>:<fire unix>, drift:<stack>:<hour>, dispatch_wave:<run>:<wave>, sync_installations:<fire unix> and sync_installations:start:<minute unix>. store.UpsertRepo takes a transaction-level advisory lock on the lowercased full name so two workers recording the same new repository do not race the unique index. store.SetDefaultGraph records repos.default_graph_id (the runs service calls it when a PR merges) and store.GetDefaultGraph reads it back; in the same transaction SetDefaultGraph takes the repository row with SELECT ... FOR UPDATE before reading the previous default graph, so two merges to one repository reconcile against each other's committed result rather than a shared stale predecessor, and reconciles stacks.removed_at to the change, stamping removed a stack the previous default graph listed and the new one does not, and clearing removed_at on a stack the new one lists, so the stack lists and /v1/overview, which filter on removed_at IS NULL, follow the default branch; a stack never on the default branch, such as one an open pull request proposes, is left alone. Saving a non-default graph (store.SaveGraph, from any plan run) never clears removed_at, so a stack deleted from the default branch stays removed even while an open pull request's plan still lists it; this holds only once every replica runs v0.6.0 or later, since a v0.5.0 replica's upsertStacks still clears removed_at on every save and can revive an already-removed stack during a rolling upgrade. Migration 0010_reconcile_removed_stacks repairs existing rows once, marking removed a stack absent from its repository's current default graph unless a graph newer than that default still lists it. repos.default_graph_id's foreign key to graphs is DEFERRABLE INITIALLY DEFERRED so the repos/graphs cycle survives a data-only dump restored in one transaction. ModuleConsumers and StackModules read each repository's default-branch graph, falling back to its latest graph. stacks.state_object (v1.StateObject as jsonb, with state_object_run_id and state_object_at) is where an apply or drift job of the newest run that reported one found the stack's state; only store.SetStackStateObject writes it, and store.ClearStackStateObject clears it for a report the run service may not record, both for apply and drift runs of the stack's repository, ignoring a late report from an older run. Saving a graph, which rewrites the PR-writable stacks.backend and stacks.config, leaves it alone. stack_resources holds one row per stack with its latest resource list as jsonb; run_id is ON DELETE SET NULL, so the list outlives its run. Only store.SetStackResources writes it, for apply and drift runs of the stack's repository (ErrInvalid otherwise, ErrNotFound for an unknown stack or run), under the observed_at rule of the runner endpoints: the caller passes a store.ResourceObservation, ObservedNow, ObservedAtStackFinish or ObservedAtRunCreation (ErrInvalid for any other), and the store computes the time in the same statement, on the database clock, always using created_at for a drift run. digest is the SHA-256 of the canonical JSON of total, truncated and the sorted items; on an unchanged digest the old resources datum and changed_at are kept and the other columns updated. store.StackResourceChanges derives each item's last change in one query, storing nothing: DISTINCT ON (address) over the stack's run_stacks rows with mode = 'apply' AND status = 'applied' (partial index run_stacks_applied_idx), expanding the summary's added, changed, replaced and destroyed, newest run first (runs.created_at DESC, runs.id DESC), with create, update and replace before destroy within a run; an address whose winner is a destroy gets no entry. This relies on summaries never being pruned: the prune job clears plan text and keeps them.

With an artifact bucket, clearing a stack's plan_text also clears its plan_url and queues the stack's plan.txt and plan.json objects in the object_deletions table in the same statement; a row an older server left with a plan_url after clearing only its plan text is reclaimed the same way, so an upgrade or a mixed-version prune strands nothing. Deleting a repository or an installation deletes its stack rows explicitly and queues the objects from the deleted rows, so a plan_url a concurrent result records up to the moment of deletion is captured rather than escaping through the cascade; object_deletions has no foreign key to the run, so it outlives it. The prune job then sweeps the queue, deleting each object through artifacts.S3Store.Delete and clearing the row; a delete that fails, such as when the role lacks s3:DeleteObject, is logged and left for the next sweep rather than failing the prune, and the sweep attempts each object once per pass, so a failure left at the head of the queue is not re-attempted until the next sweep. store.CountObjectDeletions feeds the pending_object_deletions gauge. Two instances sweeping at once only repeat a harmless delete, since the object delete is idempotent and the row is cleared after it.

CLI environment ​

VariablePurpose
STACKORDER_SERVER_URLServer base URL; unset means local mode. The CLI sets it for the tool and the hooks to the --server value, empty in local mode
STACKORDER_API_KEYAutomation key for local apply, unlock and result posting outside Actions
STACKORDER_OIDC_AUDIENCEAudience requested for the runner token; a set value is used as given (surrounding whitespace trimmed, any trailing slash kept), else the server URL with any trailing slash removed
STACKORDER_RUN_IDRun id from the resolve step or the dispatch input. The CLI sets it for the tool and the hooks to the run id it uses, empty when there is none
STACKORDER_REPOSITORYNot read. The CLI sets it for the tool and the hooks to owner/repo, from GITHUB_REPOSITORY, the event payload or the origin remote; empty when none tells it
STACKORDER_TOOL, STACKORDER_TOOL_VERSIONOverride the configured tool
STACKORDER_TERRAFORM_BIN, STACKORDER_TOFU_BINBinary name on PATH or path, overriding detection
STACKORDER_BACKEND_CONFIGExtra -backend-config values, comma separated, for init, after the stack's backend_config
STACKORDER_PLAN_DIRWhere plan files are written, default .stackorder/plans
STACKORDER_LOG_FORMATjson switches the CLI's text logs to JSON
GITHUB_*, ACTIONS_ID_TOKEN_REQUEST_URL, ACTIONS_ID_TOKEN_REQUEST_TOKEN, GITHUB_OUTPUT, GITHUB_STEP_SUMMARY, GITHUB_TOKENProvided by the runner; GITHUB_TOKEN is only used for neutral fallback checks

Exit codes: 0 success, 1 error, 2 plan has changes (affected, drift), 3 refused by the server (gate, lock, unconfirmed, fail closed).

CLI commands ​

Global flags: --server (STACKORDER_SERVER_URL), --repo-root (default GITHUB_WORKSPACE or the git top level), --verbose, --format (text, json, dot where it applies). In Actions the CLI reads the repository, SHA, event, PR number, run id and attempt from the GITHUB_* variables and the event payload; for pull_request events the run SHA is the PR head SHA from the payload, not GITHUB_SHA. Outputs go to GITHUB_OUTPUT and a Markdown summary to GITHUB_STEP_SUMMARY.

CommandFlagsOutputs
resolve--base <ref> (default: PR base SHA, then the push's before, then the merge base with origin/<default branch>), --stacks a,brun-id, matrix, waves, affected (JSON array of keys), count, unconfirmed
plan --stack <key>--run-id, --out <file>has-changes, plan-file (absolute), artifact, summary, unconfirmed
apply --stack <key> --run-id <id>--plan-file, --localsummary
drift --stack <key>--run-iddrifted, summary
check --stack <key> --run-id <id> --name <n> --status pass|fail|warn--summary, --details-url, --details-file
graph--format
affected --base <ref>--format
unlock <key>…--reason, --force-state
version

apply compares the run's SHA with the dispatch sha input (falling back to GITHUB_SHA) and with the checkout's HEAD, never with GITHUB_SHA alone, and refuses unless the stack row's lock (store.RunDetail fills it with the stack's current lock) names the run. With --plan-file missing (an expired artifact) it plans again and applies only when the new plan's resource address set matches the recorded summary; apply.from_plan: false always re-plans and compares. Outside Actions, plan and drift post results only when a server, a run id and an API key are all set.

Conventions ​

  • Errors wrap with %w and a short context; sentinel errors are exported variables named ErrX, except oidc.ErrBinding, a struct type matched with errors.As. HTTP handlers map errors to v1.Error codes in one place.
  • Every exported type and function has a doc comment. Code comments are otherwise avoided; rationale goes in commit messages.
  • All times are UTC timestamptz. JSON times are RFC 3339.
  • Handlers and workers take context.Context first and are idempotent.
  • Tests are table driven where there is a table. Integration tests carry the integration build tag; end-to-end tests carry e2e. go test ./... must pass with no Docker, no network and no Postgres.
  • Commits follow Conventional Commits with a scope: feat(graph): …, fix(cli): …, test(store): …, docs(site): …, chore(deps): …, ci: …, build(docker): ….

This page is rendered from ARCHITECTURE.md at the repository root. Change it there, in the same commit as the code it describes.