Skip to content

Metrics and tracing ​

The server exposes Prometheus metrics at GET /metrics and exports OpenTelemetry traces when an OTLP endpoint is configured. Logs are structured JSON by default.

Metric names ​

Every metric is defined in internal/metrics and carries the stackorder_ prefix. Label values that could come from outside, such as event names or HTTP methods, are bounded: a value outside the known set is counted as other (OTHER for methods). The installation label on stackorder_github_rate_limit_remaining is the exception: it is recorded verbatim, so its cardinality is unbounded.

Runs and stacks ​

MetricTypeLabelsMeaning
stackorder_runs_created_totalcountertrigger, modeRuns created, by trigger and mode; incremented once per run row, at every trigger: pull_request (a pull request's plan, and an on_merge apply after the merge), comment (a plan or apply comment), rerequest (a re-run from a check run or POST /v1/runs/{id}/rerun), push (a cross-repository plan of the stacks that depend on an upstream apply), manual (a manual apply) and schedule (a scheduled drift check)
stackorder_runs_totalcounterstatus, trigger, modeEach status a run enters, by trigger and mode; a run contributes to several series over its life, so this counts status transitions, not runs. Count runs with stackorder_runs_created_total
stackorder_stack_finished_totalcountermode, statusStacks reaching a terminal status within a run: applied, planned for plan runs, failed, blocked, noop, unconfirmed, unknown, skipped
stackorder_stack_duration_secondshistogrammodeDuration of each stack's plan, apply or drift job, from the row's start and finish times or the CLI's reported duration_ms; buckets from 5 s to 2 h
stackorder_dispatches_totalcountermode, resultworkflow_dispatch calls to stackorder-run.yml; result is ok or error
stackorder_drifted_stacksgaugeStacks whose latest drift check found drift
stackorder_locks_heldgaugeOrchestration locks held
stackorder_pending_object_deletionsgaugePlan objects waiting to be removed from the artifact bucket after the last prune; stays above zero when the server's role lacks s3:DeleteObject
stackorder_commands_totalcounterverb, acceptedPull request comment commands; verb is plan, apply, unlock, help or other, accepted is true or false

stackorder_drifted_stacks, stackorder_locks_held and stackorder_pending_object_deletions are read from the database by the instance that records a drift result, takes or releases a lock, or last ran the prune job. An instance that has not done so since it started reports 0, so aggregate them with max() across instances.

Artifact store ​

These appear only when STACKORDER_ARTIFACT_BUCKET is set; see Server configuration.

MetricTypeLabelsMeaning
stackorder_artifact_operations_totalcounteroperation, resultReads and writes against the artifact store; operation is write or read, result is ok or error

The server writes a stack's full plan text and plan JSON to the bucket and reads the plan text back to serve GET /v1/runs/{id}/stacks/{key}/plan. A failed write (operation="write",result="error") marks any bucket Put that failed, of the plan text or the plan JSON, which a broken bucket policy or credentials cause silently; only a failed plan-text write makes the server fall back to keeping the truncated plan text in Postgres, while a failed plan-JSON write still serves the full plan text from the bucket. A read of an object that is gone counts as ok, as the store answered. Compute an error rate with sum by (operation) (rate(stackorder_artifact_operations_total{result="error"}[5m])) / sum by (operation) (rate(stackorder_artifact_operations_total[5m])); aggregating the result label away on both sides keeps the division from matching the error series against itself.

Webhooks and the queue ​

MetricTypeLabelsMeaning
stackorder_webhook_received_totalcountereventDeliveries whose signature verified, by event
stackorder_webhook_duplicates_totalcounterDeliveries whose id was already queued
stackorder_webhook_lag_secondshistogramTime from receiving a webhook to a worker first starting on it; buckets from 10 ms to 10 min
stackorder_events_processed_totalcounterkind, resultQueued webhook events processed; result is ok, error (retried), dead or ignored (no handler)
stackorder_events_dead_totalcounterkindEvents given up after their last attempt
stackorder_jobs_processed_totalcounterkind, resultQueued jobs processed, by job kind, with the same results
stackorder_queue_depthgaugequeueDue rows no worker has claimed yet, for events and jobs; refreshed once a minute
stackorder_scheduler_leadergauge1 while this instance holds the scheduler's advisory lock

A failed event or job is retried after 1 min, 5 min, 30 min and 2 h, and becomes a dead letter after 5 attempts. Job kinds are dispatch_wave, drift_stack, schedule_drift, cross_repo_plan, reconcile, prune, stale_locks and sync_installations.

GitHub API ​

MetricTypeLabelsMeaning
stackorder_github_requests_totalcounterroute, statusGitHub API request attempts; route is METHOD /template, such as POST /repos/{owner}/{repo}/check-runs, and status is the HTTP status, 0 when no response arrived
stackorder_github_request_duration_secondshistogramrouteLatency of each attempt; buckets from 50 ms to 30 s
stackorder_github_rate_limit_remaininggaugeinstallationX-RateLimit-Remaining of the last GitHub API response drawn from that budget; installation is the installation id, or app for the App JWT. Absent until the first response, and so never zero on a freshly started server. Requests made with a user access token (the sign-in flow) are not tracked

HTTP and build ​

MetricTypeLabelsMeaning
stackorder_http_requests_totalcounterroute, method, statusRequests served; route is the route pattern, such as POST /v1/runs/{id}/graph, or / for the UI
stackorder_http_request_duration_secondshistogramroute, methodRequest latency, with the Prometheus default buckets
stackorder_build_infogaugeversion, commitAlways 1; the labels carry the build

The Go runtime and process collectors of prometheus/client_golang are registered too, as go_* and process_*. Labelled metrics, including github_rate_limit_remaining, appear only once a value has been observed for a label set.

Scraping ​

yaml
scrape_configs:
  - job_name: stackorder
    scheme: https
    metrics_path: /metrics
    static_configs:
      - targets: ["stackorder.example.com"]

With more than one instance, scrape each instance directly rather than through the load balancer, so every instance's counters are collected. GET /metrics is open unless STACKORDER_METRICS_TOKEN is set, in which case the scrape needs Authorization: Bearer <token>; see Security hardening for the Prometheus configuration.

Alerts worth having ​

yaml
groups:
  - name: stackorder
    rules:
      - alert: StackorderStacksDrifted
        expr: max(stackorder_drifted_stacks) > 0
        for: 1h
      - alert: StackorderWebhookLag
        expr: histogram_quantile(0.95, sum by (le) (rate(stackorder_webhook_lag_seconds_bucket[10m]))) > 60
        for: 10m
      - alert: StackorderGitHubRateLimitLow
        expr: min by (installation) (stackorder_github_rate_limit_remaining{installation!=""}) < 500
        for: 5m
      - alert: StackorderQueueBacklog
        expr: max by (queue) (stackorder_queue_depth) > 100
        for: 15m
      - alert: StackorderDeadLetters
        expr: sum(increase(stackorder_events_dead_total[1h])) > 0
      - alert: StackorderDispatchErrors
        expr: sum(increase(stackorder_dispatches_total{result="error"}[15m])) > 0
      - alert: StackorderArtifactWriteErrors
        expr: sum(increase(stackorder_artifact_operations_total{operation="write",result="error"}[15m])) > 0
      - alert: StackorderNoSchedulerLeader
        expr: max(stackorder_scheduler_leader) < 1
        for: 10m

StackorderGitHubRateLimitLow matches installation!="" so that a v0.5.0 replica still running during a rolling upgrade, which exports the gauge unlabelled and at 0 while idle, does not group under an empty installation and trip the alert; once every replica is on v0.6.0 or later the matcher is a no-op.

Also alert on /readyz: it fails when the database is unreachable.

Tracing ​

Tracing uses OpenTelemetry with the OTLP HTTP exporter. It is enabled only when OTEL_EXPORTER_OTLP_ENDPOINT is set, to the collector's base URL; spans are sent to <endpoint>/v1/traces. The exporter's other standard OTEL_EXPORTER_OTLP_* variables, such as headers, still apply.

sh
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
SpanNameAttributes
HTTP requestThe route pattern, such as POST /v1/runs/{id}/graphhttp.route; stackorder.run_id on routes under /v1/runs/{id}
Webhook eventevent <name>, such as event pull_requeststackorder.event, stackorder.delivery
Jobjob <kind>, such as job dispatch_wavestackorder.job, stackorder.job_id, and stackorder.run_id when the job belongs to a run

The resource's service.name is stackorder-server, with the build version as service.version. W3C trace context and baggage are propagated from incoming requests. /healthz, /readyz and /metrics are not traced.

A webhook delivery and the worker that later handles it, and a job and the worker that runs it, are one trace: the trace context of the request or job that queued a row is stored with it and continued when the row is claimed, so the worker span is a child of the span that queued the work and shares its trace id. A retry is a new child span under the same trace, and a span that starts long after its parent ended shows the queue delay. Rows queued before v0.6.0, and rows queued while tracing is off, carry no context, and their worker spans are roots as before.

Logs ​

The server logs with log/slog to standard error: JSON by default, text with STACKORDER_LOG_FORMAT=text. STACKORDER_LOG_LEVEL sets the level, info by default. Every request is logged once, as http request, with its request_id (also returned in X-Request-Id), route, status, duration and the kind of caller (oidc, apikey, session or none).

When tracing is on, every log record written inside a span, the access log and the event and job handlers included, also carries trace_id and span_id, so a log line can be matched to its trace and, through the stored trace context, to the request or job that queued the work. Records written outside any span, such as start-up and shutdown lines, carry neither.