Skip to content

Deploy on AWS ​

The repository ships a Terraform module in deploy/terraform that deploys the server on ECS Fargate behind an Application Load Balancer, with PostgreSQL on RDS or Aurora Serverless v2. It works with Terraform 1.11 or later and OpenTofu 1.11 or later, needs the AWS provider 6.50 or later, and Stackorder uses it to deploy itself, so it is also a working example of a stack Stackorder can manage.

hcl
module "stackorder" {
  source = "github.com/stackorder/stackorder//deploy/terraform?ref=v0.6.0"

  domain_name     = "stackorder.example.com"
  route53_zone_id = "Z0123456789ABCDEFGHIJ"
  image_tag       = "0.6.0"
}

Set exactly one of route53_zone_id, to have the module issue a DNS-validated certificate and create the alias record, or certificate_arn, to bring a certificate and point your own DNS at alb_dns_name. deploy/terraform/examples/ has three complete calls: complete (a new VPC, the artifact bucket and alarms), existing-vpc (your VPC and certificate) and self-hosted (the stack through which Stackorder deploys and upgrades itself).

What the module creates ​

ResourcePurpose
VPC with public and private subnets and NAT, unless create_vpc = falseTwo availability zones by default; one NAT gateway unless single_nat_gateway = false, and none with public_tasks = true
Application Load Balancer, HTTPS listener with TLS 1.3, HTTP redirectForwards to port 8080; health checks on /readyz; 30 s deregistration delay; access logs to an S3 bucket (alb_access_logs_enabled), a WAF web ACL (waf_web_acl_arn) and single sign-on (oidc_authentication or cognito_authentication) are optional
ACM certificate and Route53 records, with route53_zone_idThe certificate and alias for domain_name
ECS cluster and Fargate service, 1 or 2 tasksRuns ghcr.io/stackorder/stackorder as user 65532, read-only root file system, with the deployment circuit breaker; the container health check runs stackorder-server healthcheck
RDS PostgreSQL 17 (db.t4g.micro, gp3, encrypted), or Aurora Serverless v2The database, with rds.force_ssl = 1, 7 days of point-in-time recovery, deletion protection and a final snapshot
Three Secrets Manager secretsDATABASE_URL; a JSON secret with the App credentials, STACKORDER_SESSION_KEY and STACKORDER_METRICS_TOKEN, injected through ECS secrets; and a copy of the metrics token for scrapers
Execution role and task roleThe task role has no permissions unless the artifact bucket, shared outputs or ECS Exec is enabled
Security groupsLoad balancer: 80 and 443 from ingress_cidrs, 8080 to the tasks, and 443 out to alb_https_egress_cidrs, or anywhere with single sign-on. Tasks: 8080 from the load balancer, 443 out, Postgres to the database. Database: Postgres from the tasks
CloudWatch log groupThe server's logs, 30 days by default
Artifact bucket, with artifact_bucket_enabledFull plan text, deleted by the server when its plan text is pruned or its repository removed, and expiring after artifact_retention_days as a backstop
CloudWatch alarms, with alarms_enabledTarget 5xx, unhealthy targets, CPU, and database free storage or ACU utilization

The server needs nothing else. It holds no AWS credentials for your infrastructure; with shared outputs on, it can read the state objects you list, or, with shared_outputs_role_arn, whatever that role allows, so give the role only those reads.

Security groups filter by address, not by host name, so the tasks' egress is HTTPS to anywhere, through NAT: the tasks pull the image from ghcr.io and reach the GitHub API, GitHub's OIDC keys, Secrets Manager and CloudWatch Logs over their public endpoints. For stricter egress, put a proxy or firewall on the NAT path.

For a single-user deployment the NAT gateway is the largest fixed cost after the database, so public_tasks = true runs the tasks in the public subnets with public IP addresses instead, and a created VPC gets no NAT gateway. The service security group still admits only the load balancer, so nothing can connect to the tasks' public addresses, and the database stays in the private subnets. With an existing VPC the tasks then use public_subnet_ids, and private_subnet_ids stays required for the database.

Inputs ​

The tables follow the descriptions in deploy/terraform/variables.tf and deploy/terraform/outputs.tf, and docs/test fails when they drift apart. Every input also has validation rules, which the descriptions summarise.

Naming ​

InputTypeDefaultDescription
namestring"stackorder"Name of the deployment, used as the name or prefix of every resource. Lowercase letters, digits and single hyphens, 2 to 24 characters.
tagsmap(string){}Tags added to every resource that supports them.

Network ​

InputTypeDefaultDescription
create_vpcbooltrueCreate a VPC with public and private subnets and, unless public_tasks is true, NAT. When false, vpc_id, public_subnet_ids and private_subnet_ids are required.
vpc_cidrstring"10.0.0.0/16"IPv4 CIDR of the VPC created when create_vpc is true. Subnets are carved as eight equal blocks: public from the first four, private from the last four.
vpc_idstringnullID of an existing VPC. Required when create_vpc is false, must be null otherwise.
public_subnet_idslist(string)[]IDs of existing public subnets in at least two availability zones, for the load balancer and, when public_tasks is true, the tasks. Required when create_vpc is false.
private_subnet_idslist(string)[]IDs of existing private subnets in at least two availability zones, for the database and, unless public_tasks is true, the tasks, which then need a route to the internet through NAT. Required when create_vpc is false.
availability_zoneslist(string)[]Availability zones for the created VPC. Empty picks the first two available zones of the region.
single_nat_gatewaybooltrueUse one NAT gateway and route table for all private subnets instead of one per availability zone. Without NAT, when public_tasks is true, it only sets the number of private route tables.
public_tasksboolfalseRun the tasks in the public subnets with public IP addresses instead of in the private subnets behind NAT, so a created VPC needs no NAT gateway. The service security group still admits only the load balancer; the database stays in the private subnets.

Name, TLS and ingress ​

InputTypeDefaultDescription
domain_namestringrequiredFully qualified host name the server is reached at, such as stackorder.example.com.
route53_zone_idstringnullRoute53 hosted zone in which to create the ACM validation records and the alias record for domain_name. Exactly one of route53_zone_id and certificate_arn must be set.
certificate_arnstringnullARN of an existing ACM certificate covering domain_name. DNS for domain_name is then left to the caller. Exactly one of route53_zone_id and certificate_arn must be set.
base_urlstringnullPublic URL of the server when it differs from https://<domain_name>, for example behind another proxy. No trailing slash.
ssl_policystring"ELBSecurityPolicy-TLS13-1-2-2021-06"Security policy of the HTTPS listener. Must be a TLS 1.3 policy.
ingress_cidrslist(string)["0.0.0.0/0"]IPv4 or IPv6 CIDRs allowed to reach the load balancer on ports 80 and 443. GitHub webhooks and GitHub-hosted runners need the default. Ignored when github_webhook_ip_ranges_only is true.
github_webhook_ip_ranges_onlyboolfalseRestrict the load balancer to GitHub's webhook source ranges (the hooks list of the GitHub meta API, read at plan time) plus admin_cidrs, instead of ingress_cidrs.
admin_cidrslist(string)[]CIDRs of people and self-hosted runners that need the UI and API when github_webhook_ip_ranges_only is true.
alb_access_logs_enabledboolfalseWrite load balancer access logs to an S3 bucket the module creates, encrypted with SSE-S3 as ELB log delivery requires.
alb_access_logs_retention_daysnumber90Days after which objects in the access log bucket expire.
waf_web_acl_arnstringnullARN of a regional AWS WAFv2 web ACL in the module's region to associate with the load balancer. Null associates none.
oidc_authenticationobject (sensitive)nullOpenID Connect provider with which the load balancer authenticates people before forwarding, as the authenticate_oidc action of the HTTPS listener. Webhooks, health checks, and runner, CLI and metrics requests that carry a bearer token bypass it. The listener stores client_secret in Terraform state. Null authenticates nobody at the load balancer.
cognito_authenticationobjectnullAmazon Cognito user pool with which the load balancer authenticates people before forwarding, as the authenticate_cognito action of the HTTPS listener, with the same bypass as oidc_authentication. At most one of oidc_authentication and cognito_authentication may be set. Null authenticates nobody at the load balancer.
alb_https_egress_cidrslist(string)[]IPv4 CIDRs the load balancer may reach on port 443, as it must to reach the identity provider of oidc_authentication or cognito_authentication. Empty allows none, or 0.0.0.0/0 when either authentication is set, because identity providers publish no fixed address ranges.

Service ​

InputTypeDefaultDescription
imagestring"ghcr.io/stackorder/stackorder"Container image repository of the server.
image_tagstring"latest"Tag or digest (sha256:...) of the server image. Pin a release such as 1.2.3 so upgrades are explicit plans.
verify_imagebooltrueWhen image is on ghcr.io, check at plan time that image_tag can be pulled anonymously, as ECS pulls it, and fail the plan if not, instead of letting ECS retry the pull until the deployment times out. Needs HTTPS access to ghcr.io from where Terraform runs.
desired_countnumber1Number of server tasks, but one task while the GitHub App inputs are unset, so there is a single setup token. All coordination goes through Postgres, so a second task adds availability without any other change.
cpunumber256Fargate task CPU units.
memorynumber512Fargate task memory in MiB; must be a valid combination with cpu.
cpu_architecturestring"X86_64"CPU architecture of the task, X86_64 or ARM64.
create_clusterbooltrueCreate an ECS cluster for the service. When false, cluster_arn names an existing cluster to run in.
cluster_arnstringnullARN of an existing ECS cluster to run the service in. Required when create_cluster is false, must be null otherwise. With a capacity_provider_strategy the cluster must already have those capacity providers associated, which the module does only for a cluster it creates.
capacity_provider_strategylist(object[]Capacity provider strategy for the service, which replaces launch_type = FARGATE; set it to run tasks on FARGATE_SPOT. Each entry weights a provider, FARGATE or FARGATE_SPOT, and may set a base, the first number of tasks to place on it. At least one provider must have a positive weight, as ECS places no tasks when every weight is zero and an omitted weight is zero. Empty keeps launch_type = FARGATE. For a created cluster the module associates the named providers with it; an existing cluster must already have them.
enable_execute_commandboolfalseEnable ECS Exec. The SSM agent needs a writable root file system, so this also turns readonlyRootFilesystem off and grants the task role the ssmmessages permissions.
health_check_commandlist(string)["CMD", "/stackorder-server", "healthcheck"]Container health check command, starting with CMD or CMD-SHELL. The default runs the server's healthcheck subcommand, which GETs /healthz on the listen port. The distroless image has no shell or curl, so a replacement must be a command the image itself provides. Empty turns the container health check off and leaves task health to the load balancer check on /readyz.
stop_timeout_secondsnumber60Seconds ECS waits after SIGTERM before it kills the container (stopTimeout), 2 to 120 on Fargate. The server drains HTTP for up to 15 s, then its workers for up to 30 s plus 5 s for cancelled handlers, so a value under 50 can cut the drain short.
wait_for_steady_statebooltrueMake terraform apply wait until the new tasks pass /readyz, so an apply of an upgrade fails when the deployment rolls back.
deployment_timeoutstring"20m"How long terraform apply waits for the service to reach a steady state when wait_for_steady_state is true, as the create and update timeout of the ECS service, such as 20m or 1h.
log_retention_daysnumber30Retention of the server log group in days.
log_levelstring"info"Server log level (STACKORDER_LOG_LEVEL).
extra_environmentmap(string){}Additional environment variables for the server, such as STACKORDER_WORKERS or OTEL_EXPORTER_OTLP_ENDPOINT. Variables the module sets itself are rejected.

Database ​

InputTypeDefaultDescription
engine_versionstring"17"PostgreSQL major version, or major.minor. For Aurora a major version resolves to the AWS default minor of that major at plan time.
allow_major_version_upgradeboolfalseAllow a new major version in engine_version to upgrade the RDS instance or Aurora cluster in place. A major upgrade cannot be rolled back; take a snapshot first and set apply_immediately for the same apply.
apply_immediatelyboolfalseApply database changes, such as engine_version, instance_class or the parameter group, at once instead of in the next maintenance window. Changes that need a restart then cause a short outage.
instance_classstring"db.t4g.micro"RDS instance class. Ignored when use_aurora_serverless is true.
allocated_storagenumber20Initial RDS storage in GiB. Ignored when use_aurora_serverless is true.
max_allocated_storagenumber100Upper bound for RDS storage autoscaling in GiB; 0 disables autoscaling. Ignored when use_aurora_serverless is true.
multi_azboolfalseRun the RDS instance Multi-AZ, or add an Aurora reader in another zone.
deletion_protectionbooltrueProtect the database from deletion.
skip_final_snapshotboolfalseSkip the final database snapshot on destroy. Keep false outside of throwaway environments.
backup_retention_daysnumber7Automated backup retention in days; point-in-time recovery covers this window.
performance_insightsboolfalseEnable Performance Insights with the free 7 day retention. Not every instance class supports it.
use_aurora_serverlessboolfalseUse an Aurora PostgreSQL Serverless v2 cluster instead of an RDS instance. Switching an existing deployment replaces the database.
aurora_min_acunumber0.5Minimum Aurora Serverless v2 capacity in ACUs.
aurora_max_acunumber2Maximum Aurora Serverless v2 capacity in ACUs.
kms_key_arnstringnullCustomer managed KMS key for the Secrets Manager secrets and database storage. Null uses the AWS managed keys.

GitHub App and server settings ​

InputTypeDefaultDescription
github_app_idstringnullGitHub App id (GITHUB_APP_ID). Leave the App inputs null on the first deploy: the server starts in setup mode and /setup creates the App.
github_app_private_keystring (ephemeral, sensitive)nullPEM private key of the GitHub App (GITHUB_APP_PRIVATE_KEY).
github_webhook_secretstring (ephemeral, sensitive)nullWebhook secret of the GitHub App (GITHUB_WEBHOOK_SECRET).
github_oauth_client_idstringnullOAuth client id of the GitHub App, for human sign-in (GITHUB_OAUTH_CLIENT_ID).
github_oauth_client_secretstring (ephemeral, sensitive)nullOAuth client secret of the GitHub App (GITHUB_OAUTH_CLIENT_SECRET).
session_keystring (ephemeral, sensitive)null32 byte hex key for cookie signing (STACKORDER_SESSION_KEY). Null generates a new one whenever the app secret is written, which ends every session.
metrics_tokenstring (ephemeral, sensitive)nullBearer token that GET /metrics requires (STACKORDER_METRICS_TOKEN), at least 16 printable ASCII characters without white space. Null generates 32 hexadecimal characters whenever the app secret is written.
db_password_versionnumber1Version of the generated database password. Increase it to rotate the password: the database and the DATABASE_URL secret get the new one in the same apply, and the service rolls.
secrets_versionnumber1Version of the app and metrics token secrets. Increase it after changing an ephemeral input, or to generate a new session key and metrics token; both secrets are rewritten and the service rolls.
secret_recovery_window_daysnumber30Days Secrets Manager keeps a deleted secret recoverable; 0 deletes immediately.
database_secret_arnstringnullARN of an existing Secrets Manager secret the module writes DATABASE_URL into, instead of creating <name>/database-url. The module owns the secret value and writes it with a write-only argument; you own the secret resource, its name, KMS key, recovery window and policies. Null creates the secret.
app_secret_arnstringnullARN of an existing Secrets Manager secret the module writes the app secret into (the GitHub App credentials, the session key and the metrics token, as JSON), instead of creating <name>/app. The module owns the secret value and writes it with a write-only argument; you own the secret resource, its name, KMS key, recovery window and policies. Null creates the secret.
metrics_token_secret_arnstringnullARN of an existing Secrets Manager secret the module writes the metrics token into as plain text, for scrapers, instead of creating <name>/metrics-token. The module owns the secret value and writes it with a write-only argument; you own the secret resource, its name, KMS key, recovery window and policies. Null creates the secret.
github_api_urlstring"https://api.github.com"GitHub API base URL (GITHUB_API_URL); GitHub Enterprise Server uses https://<host>/api/v3.
required_workflow_refstringnullGlob that runner tokens' job_workflow_ref must match (STACKORDER_REQUIRED_WORKFLOW_REF), such as stackorder/actions/.github/workflows/.yml@refs/tags/v1. Null accepts any workflow.
oidc_audiencestringnullAudience runner OIDC tokens must carry (STACKORDER_OIDC_AUDIENCE). Null uses the public URL.

Artifact bucket and alarms ​

InputTypeDefaultDescription
artifact_bucket_enabledboolfalseCreate an S3 bucket for full plan text (STACKORDER_ARTIFACT_BUCKET) and grant the task role s3:GetObject, s3:PutObject and s3:DeleteObject on it, and s3:ListBucket. It is the only bucket the task role can write to.
artifact_retention_daysnumber90Days after which objects in the artifact bucket expire. The server already deletes a plan's objects when its plan text is pruned or its repository removed, so this lifecycle rule is a backstop.
alarms_enabledboolfalseCreate CloudWatch alarms for target 5xx responses, unhealthy targets and service CPU, plus database free storage (RDS) or ACU utilization (Aurora, whose storage grows on its own).
alarm_sns_topic_arnstringnullSNS topic notified when an alarm changes state. Required when alarms_enabled is true.
alarm_thresholdsobject{}Alarm thresholds: target 5xx responses per 5 minutes, average service CPU percent, RDS free storage in bytes, and Aurora ACU utilization percent.

Shared outputs ​

InputTypeDefaultDescription
shared_outputs_statelist(object({ repository = string, bucket = string, prefix = optional(string, ""), kms_key_arn = optional(string) }))[]State objects whose root outputs the server may serve to the readers a stack lists (STACKORDER_STATE_BUCKETS): each entry grants one repository, owner/repo, or every repository with *, the objects of an S3 bucket under an optional key prefix, with the KMS key of those objects. The server reads a stack's state only under an entry of the stack's own repository, and a prefix covers a key only at a path segment boundary. Unless shared_outputs_role_arn is set, the task role gets read-only access to the objects under each prefix, for every repository at once, and kms:Decrypt through S3 with each kms_key_arn. Empty turns shared outputs off.
shared_outputs_role_arnstringnullIAM role the server assumes to read the state objects of shared_outputs_state (STACKORDER_STATE_ROLE_ARN), such as a role in the account of the state buckets. The task role then gets only sts:AssumeRole on it, and the role's own policies must grant the reads and kms:Decrypt, so shared_outputs_state takes no kms_key_arn. Null reads with the task role. Requires shared_outputs_state.
shared_outputs_external_idstringnullExternal ID the server sends when it assumes shared_outputs_role_arn (STACKORDER_STATE_EXTERNAL_ID), for a role whose trust policy requires a matching sts:ExternalId, as a role in another account commonly does. Null sends none. Requires shared_outputs_role_arn.

Reusing existing infrastructure ​

Each resource the module creates is optional through an input that defaults to creating it, so the defaults deploy a self-contained stack and an existing deployment that sets none of these inputs plans with no changes. The module moves the cluster and the three secrets under count with moved blocks, so the upgrade needs no state surgery.

  • create_vpc = false runs in an existing VPC named by vpc_id, with public_subnet_ids and private_subnet_ids (see Network and the examples/existing-vpc example).
  • create_cluster = false runs the service in the existing cluster named by cluster_arn instead of creating one; ecs_cluster_name is then the name parsed from the ARN.
  • capacity_provider_strategy runs the tasks under a capacity provider strategy in place of launch_type = FARGATE, which ECS forbids alongside it; use it to place tasks on FARGATE_SPOT. At least one provider must have a positive weight, as ECS places no tasks when every weight is zero and an omitted weight is zero. For a cluster the module creates it associates the strategy's providers; an existing cluster must already have them associated.
  • database_secret_arn, app_secret_arn and metrics_token_secret_arn make the module write each value into a Secrets Manager secret you already own instead of creating <name>/database-url, <name>/app and <name>/metrics-token. The module still composes and writes the value with a write-only argument, so nothing secret reaches state or the plan; you own the secret resource and its name, KMS key, recovery window, rotation and resource policy. The execution role is granted kms:Decrypt on kms_key_arn only, so a supplied secret encrypted with a different customer managed key must grant the execution role access in that key's policy.

An existing database or load balancer is out of scope; the module always creates those.

What the module passes to the server ​

VariableSource
STACKORDER_BASE_URLbase_url, or https://<domain_name>
STACKORDER_LISTEN:8080
GITHUB_API_URLgithub_api_url
STACKORDER_OIDC_AUDIENCEoidc_audience, or the base URL
STACKORDER_REQUIRED_WORKFLOW_REFrequired_workflow_ref, when set
STACKORDER_ARTIFACT_BUCKETThe artifact bucket, when artifact_bucket_enabled
STACKORDER_STATE_BUCKETSshared_outputs_state, as comma-separated repository=bucket/prefix entries, or repository=bucket without a prefix, when it is not empty
STACKORDER_STATE_ROLE_ARNshared_outputs_role_arn, when set
STACKORDER_STATE_EXTERNAL_IDshared_outputs_external_id, when set
STACKORDER_LOG_LEVELlog_level
DATABASE_URLThe database secret: postgres://stackorder:<password>@<endpoint>:5432/stackorder?sslmode=require
GITHUB_APP_ID, GITHUB_APP_PRIVATE_KEY, GITHUB_WEBHOOK_SECRET, GITHUB_OAUTH_CLIENT_ID, GITHUB_OAUTH_CLIENT_SECRETThe App secret, when set
STACKORDER_SESSION_KEYThe App secret; session_key, or generated
STACKORDER_METRICS_TOKENThe App secret; metrics_token, or generated. Scrapers read the same value from the <name>/metrics-token secret

Everything else the server reads, such as STACKORDER_WORKERS, the retention durations, STACKORDER_LOG_FORMAT, GITHUB_WEB_URL, GITHUB_OIDC_ISSUER or OTEL_EXPORTER_OTLP_ENDPOINT, goes through extra_environment, which rejects the variables above. Values in extra_environment are plain text in the task definition, so keep secrets out of it; the metrics token has its own input, metrics_token, and scrapers read it from the secret named by the metrics_token_secret_arn output, see Security hardening. On GitHub Enterprise Server set github_api_url and add GITHUB_WEB_URL and GITHUB_OIDC_ISSUER to extra_environment.

Container health check ​

By default the container runs stackorder-server healthcheck every 30 s with a 5 s timeout, 3 retries and a 30 s start period. It asks /healthz, which does not touch the database, so the container check fails only when the process itself stops answering. The load balancer's /readyz check, database included, gates deployments and drives the circuit breaker, and ECS also replaces a task that fails it, so a database outage still makes ECS replace tasks, whatever health_check_command is. A replacement task then exits at start-up until the database answers again. Set health_check_command = [] to turn the container check off. stop_timeout_seconds (60 by default) gives the server time to drain HTTP and its workers before ECS sends SIGKILL.

Outputs ​

OutputDescription
urlPublic URL of the server (STACKORDER_BASE_URL).
alb_dns_nameDNS name of the load balancer; point a CNAME or alias here when DNS is not managed by the module.
alb_zone_idRoute53 zone id of the load balancer, for alias records.
setup_urlPage that creates the GitHub App from a manifest on the first deploy. It opens only with the one-time token the server logs at start-up: take the full URL from the setup_url line in the log_group_name log group.
webhook_urlWebhook URL of the GitHub App.
ecs_cluster_nameName of the ECS cluster the service runs in, whether the module created it or create_cluster is false.
ecs_service_nameName of the ECS service.
task_definition_arnARN of the current task definition revision.
db_endpointHost name of the database writer endpoint.
db_secret_arnARN of the Secrets Manager secret holding DATABASE_URL, whether the module created it or database_secret_arn is set.
app_secret_arnARN of the Secrets Manager secret holding the GitHub App credentials, the session key and the metrics token as JSON, whether the module created it or app_secret_arn is set.
metrics_token_secret_arnARN of the Secrets Manager secret holding only the /metrics bearer token, as plain text, whether the module created it or metrics_token_secret_arn is set; grant Prometheus read access to this one rather than to the app secret.
artifact_bucketName of the artifact bucket, or null when artifact_bucket_enabled is false.
alb_access_logs_bucketName of the load balancer access log bucket, or null when alb_access_logs_enabled is false.
security_group_idsSecurity group ids of the load balancer, the service and the database.
alb_arnARN of the load balancer.
https_listener_arnARN of the HTTPS listener, for listener rules of your own at priority 100 or above; the module keeps priorities 1 to 99 for its rules.
http_listener_arnARN of the HTTP listener, which redirects to HTTPS.
target_group_arnARN of the target group of the server's tasks, for listener rules that forward to the server.
log_group_nameCloudWatch log group of the server.

Single sign-on in front of the UI ​

Set oidc_authentication for any OpenID Connect provider, such as Google Workspace, Microsoft Entra ID or Okta, or cognito_authentication for a Cognito user pool, and the HTTPS listener authenticates people before the server sees them: its default action becomes authenticate, then forward. The UI, /setup and the server's own GitHub sign-in (/auth/login and /auth/callback) stay behind it. People still get their Stackorder identity and permissions from GitHub, because the server does not read the load balancer's x-amzn-oidc-* headers.

Machines cannot sign in interactively, so the module adds listener rules that forward without authentication:

PriorityPathOther conditionsCallers
1/webhooks/githubPOSTGitHub webhook deliveries, verified by their signature
2/healthz, /readyzGET or HEADUptime checks and external monitoring
3/v1/runs, /v1/runs/*Authorization: Bearer *Actions jobs with their OIDC token, and the CLI with an OIDC token or an API key
4/v1/unlock, /v1/meAuthorization: Bearer *The CLI with an OIDC token or an API key
5/metricsGET, Authorization: Bearer *Prometheus with the metrics token

The server still authenticates every request these rules let through. The UI's own /v1 calls carry a session cookie rather than a bearer token, so they stay behind the load balancer, and scripts that call other /v1 endpoints with an API key need a rule of their own. The module keeps priorities 1 to 99; attach your own rules to the https_listener_arn output at 100 or above.

The load balancer calls the identity provider itself, so it needs HTTPS egress. Identity providers publish no fixed address ranges, so with authentication set and alb_https_egress_cidrs empty the module allows 443 to 0.0.0.0/0; set alb_https_egress_cidrs to narrow it. The AWS provider has no write-only argument for the OIDC client secret, so the listener stores client_secret in Terraform state. With Google Workspace, create a "Web application" OAuth client with the redirect URI https://<domain_name>/oauth2/idpresponse and set the consent screen's user type to Internal, which is what limits sign-in to your Workspace. deploy/terraform/README.md has a complete example.

Shared outputs ​

Shared outputs let a stack read another stack's root outputs through the server, so its own role needs no access to the other stack's state. The server then reads the producer's state object with its own credentials, which the module grants only when shared_outputs_state lists where the state of each repository lives:

hcl
shared_outputs_state = [
  {
    repository  = "acme/infra"
    bucket      = "acme-tfstate"
    prefix      = "acme/infra/"
    kms_key_arn = "arn:aws:kms:eu-west-1:123456789012:key/00000000-0000-0000-0000-000000000000"
  },
  {
    repository  = "acme/platform"
    bucket      = "acme-tfstate"
    prefix      = "acme/platform/"
    kms_key_arn = "arn:aws:kms:eu-west-1:123456789012:key/00000000-0000-0000-0000-000000000000"
  },
]

The list reaches the server as STACKORDER_STATE_BUCKETS, which both turns the feature on and bounds it. Each entry belongs to one repository, owner/repo: the location an apply or drift job of a repository reports for a stack is recorded and read only when an entry of that repository covers it, so one repository cannot point its stacks at another repository's state, as long as the entries of different repositories do not overlap. The server logs a warning at start-up for each pair of entries of different repositories where one's prefix lies under the other's. repository = "*" grants an entry to every repository the server knows; use it only for state that any of them may expose. A prefix covers a key only at a path segment boundary: stacks/prod covers stacks/prod/vpc.tfstate but not stacks/production/vpc.tfstate. The module refuses a repository with any other wildcard and a prefix with a . or .. segment.

The task role gets s3:GetObject on the objects under each prefix, s3:ListBucket limited to the same prefixes, and kms:Decrypt on each kms_key_arn through S3 only. Nothing grants write or delete access. Without s3:ListBucket, S3 answers a request for a missing object with access denied rather than not found; the prefix-limited grant is there so that a missing state object can show as missing, but treat an access-denied read as either a missing object or a missing grant. IAM cannot tell repositories apart: the policy covers every entry's prefix for the server as a whole, and the server's own check of the producer's repository is what keeps each repository to its entries. The server's role can read the whole state objects under the listed buckets and prefixes, secrets included, though the server returns only their outputs, so list the narrowest prefixes that hold the state of the stacks that share outputs rather than whole buckets, one per repository, and turn on CloudTrail data events for those buckets to audit the reads. See Security model for what a compromised server can then read.

To keep these reads apart from the task role, or to read buckets in another account, create a dedicated role that has only those read permissions and trusts the task role, arn:aws:iam::<account>:role/<name>-task, then set shared_outputs_role_arn. The task role then gets only sts:AssumeRole on that role, and the server reads through it, so the role's own policies grant kms:Decrypt and the entries of shared_outputs_state take no kms_key_arn. A bucket or KMS key in another account than the reading role must also allow that role in its bucket or key policy.

When the role is in another account, its trust policy commonly requires an external ID to guard against a confused deputy. Set shared_outputs_external_id to the value its sts:ExternalId condition expects, and the module passes it to the server as STACKORDER_STATE_EXTERNAL_ID:

hcl
shared_outputs_role_arn    = "arn:aws:iam::210987654321:role/stackorder-state-reader"
shared_outputs_external_id = "acme-stackorder"

The reader role's trust policy then admits the task role only with that external ID:

json
{
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::123456789012:role/stackorder-task" },
  "Action": "sts:AssumeRole",
  "Condition": { "StringEquals": { "sts:ExternalId": "acme-stackorder" } }
}

First deployment ​

The server starts in setup mode while the GitHub App inputs are unset, serving only /setup, /healthz and /readyz. The first deployment uses that:

  1. Apply the module with the github_* inputs unset. The service comes up in setup mode, with one task whatever desired_count says, so there is a single setup token.

  2. Take the setup URL, with its one-time setup token, from the task logs:

    sh
    aws logs tail "$(terraform output -raw log_group_name)" --since 1d | grep setup_url

    The setup_url output is the same page without the token, which answers 403. Every task start generates a new token, so take the latest line. The module keeps one task until the App inputs are set because each task has its own token, and with two the load balancer could send the browser to the other one. The server logs the line once, when the task starts; to get a new one, restart the task:

    sh
    aws ecs update-service --cluster "$(terraform output -raw ecs_cluster_name)" \
      --service "$(terraform output -raw ecs_service_name)" --force-new-deployment

    With log_level = "error" the line is not logged; keep info or warn until the App exists, or set STACKORDER_SETUP_TOKEN through extra_environment and open /setup?token= followed by its value. A token set that way is plain text in the task definition and in state, so remove it once the App exists.

  3. Open that URL and create the App. The page prints the App id, private key, webhook secret and OAuth client id and secret once.

  4. Apply again with those five values. The App id, private key and webhook secret must be set together, as must the two OAuth values. Setting them rewrites the App secret without a change to secrets_version.

  5. Install the App on your repositories and continue with Getting started.

The task definition carries the version ids of both secrets as Docker labels, so a new secret version rolls the service onto it.

Secrets and Terraform state ​

This section applies from module v0.2.0. v0.1.0 keeps these values in state and has no secrets_version or db_password_version.

The module keeps the App private key, webhook secret and OAuth client secret, the session key, the metrics token and the database password out of Terraform state and saved plans, as long as the caller passes the secret inputs as ephemeral values too. The three App secrets, session_key and metrics_token are ephemeral inputs. The database password, and the session key and metrics token when those inputs are null, come from ephemeral random_password resources. They reach AWS only through write-only attributes: secret_string_wo on the secret versions and password_wo or master_password_wo on the database. State still describes the deployment, so keep it encrypted and access controlled, but it holds none of these values.

Two inputs are exceptions and stay in state and in saved plans: the client_secret of oidc_authentication, which the load balancer listener keeps (see Single sign-on), and every value passed through extra_environment.

A write-only value is sent only when its version changes, so a changed input does nothing until a version moves:

  • secrets_version rewrites the App and metrics token secrets. Increase it after changing any ephemeral input. Each rewrite generates a new session key and metrics token unless session_key and metrics_token are set, which signs everyone out and means scrapers must read the new token.
  • db_password_version generates a new database password and writes it to the database and to the DATABASE_URL secret in the same apply. Until the new tasks replace the old ones, the old tasks keep their open database connections but cannot open new ones.
  • The module rewrites the App and metrics token secrets on its own when the App id or OAuth client id changes, including when the App inputs are first set, and when either secret is recreated. It writes a new database password and DATABASE_URL when the database or its secret is created or replaced.

The generated values are new on every run; only the secrets keep them. If an apply that writes the database password or DATABASE_URL fails, do not simply run it again: the retry would write the side that failed with a different password than the side that succeeded. Increase db_password_version and apply. Likewise, increase secrets_version after a failed apply that writes the App or metrics token secret while the metrics token is generated.

A saved plan does not carry ephemeral values. Supply the ephemeral inputs when applying it as well as when planning, with TF_VAR_* variables or -var, or read them in the calling configuration with an ephemeral resource such as ephemeral "aws_secretsmanager_secret_version", which is read again at apply time, as examples/self-hosted does. Terraform refuses to apply a saved plan when an ephemeral input set at plan is missing. OpenTofu instead uses the input's default, so a session_key or metrics_token omitted at apply makes the module write new generated values in place of the chosen ones.

Under Stackorder, the App inputs reach plan jobs

Stackorder applies saved plans, so the ephemeral inputs must reach its plan jobs as well as its applies, and the module requires github_app_id, the private key and the webhook secret together at plan time. Plan jobs for a pull request run that pull request's code, so anyone who can push a branch can read the values, whether they come in as TF_VAR_* variables or through an ephemeral read with the plan role. Passing them as TF_VAR_* in the env secret of the reusable workflows also puts the App private key in a repository or organization secret, and has two more problems: an ENV environment secret replaces the whole env secret, so a production ENV secret without them drops them from applies, and the env secret needs stackorder/actions v1.1.0 or later. Read the App secrets with an ephemeral resource instead, as examples/self-hosted does, keep the stack in a repository where only its operators can push branches, and see Provider credentials.

Upgrades ​

  1. Change image_tag to the new X.Y.Z, or a sha256: digest, and apply.
  2. ECS starts a new task while the old one keeps serving (100 % minimum healthy, up to 200 % during the deployment). The new task runs any database migrations at start-up, under a migration lock.
  3. The new task receives traffic once it passes /readyz; the old one drains for 30 s and stops. A task that never becomes healthy is rolled back by the circuit breaker, and wait_for_steady_state makes the apply fail.

ECS retries a failed image pull until the deployment times out, so a private package or a tag that does not exist yet would otherwise hold the apply for the whole of deployment_timeout (20m by default). For images on ghcr.io, verify_image makes the plan fetch an anonymous pull token and the manifest of image_tag, as ECS would, and fail with an explanation unless ghcr.io answers 200. The check needs HTTPS access to ghcr.io from wherever Terraform runs; verify_image = false skips it.

Pin a specific version rather than latest, so an apply is the only thing that changes the running version. Take a database snapshot before an upgrade that includes migrations. See Upgrades and backups.

Upgrading the module from v0.1.0, which kept the secrets in state, needs Terraform or OpenTofu 1.11. In the examples/self-hosted setup, first set the new github_app_id and github_oauth_client_id variables of the stack, which it no longer reads from the App secret; otherwise the server returns to setup mode. The first apply removes the module's random_password and random_bytes resources from state, which touches nothing in AWS, writes a new database password to the database and to DATABASE_URL, and rewrites the App and metrics token secrets, so the service rolls. Until the new tasks replace the old ones, the old tasks keep their open database connections but cannot open new ones. Unless session_key and metrics_token are set, both are new: everyone signs in again, and scrapers must read the new token. Older state versions, for example in a versioned S3 bucket, still hold the values of v0.1.0. The database password, session key and generated token in them no longer work, but the App private key, webhook secret and OAuth client secret stay valid until you rotate them in the App settings.

Scaling out ​

Set desired_count = 2 for availability. No other change is needed:

  • webhook handling, workers and the API are stateless;
  • all coordination goes through Postgres, and workers claim work with SKIP LOCKED;
  • the scheduler is single-leader through a Postgres advisory lock, so one task schedules and any task executes;
  • sessions live in Postgres, and nothing on local disk matters.

Both tasks read the same STACKORDER_SESSION_KEY from the App secret. Raise cpu and memory before adding tasks, and consider db.t4g.small or Aurora once the database is the bottleneck.

Testing the module ​

sh
cd deploy/terraform
terraform init -backend=false
terraform test

The tests plan against mock AWS and HTTP providers and need no AWS account. Mock providers cannot serve ephemeral resources, so the tests use the real random provider and check the values written to the secrets through the module's locals. They need Terraform 1.11.4 or later: 1.11.0 to 1.11.3 return values for write-only attributes from mock providers, which fails the plans. CI runs them on the latest 1.11 and 1.14 releases. OpenTofu rejects the tests' mock provider syntax at init, so run them with Terraform; to validate the module with OpenTofu, copy it without the tests directory.

Managing the module with Stackorder ​

Put the module call in its own stack, with its own backend "s3" block, in a repository that has Stackorder installed, as examples/self-hosted does. Plans for the server's own changes then run like any other stack. Applies depend on the server being up, so for an upgrade that might not come back healthy, keep a way to apply the stack by hand.