Secure Software Release Pipelines

This page assembles the cluster. Every control described separately elsewhere appears here in the order it runs, with the evidence each stage produces and the stage that checks it.

The design principle throughout:

Each stage should produce evidence the next stage can verify without trusting the stage that produced it.

Before you start

• Software supply chain security — the threats
• Artifact attestations and build provenance
• Reusable workflows and environments
• Repository rulesets for security

The model

A release pipeline with verification at every handoff

A vertical chain: protected source, reviewed pull request, trusted build workflow, tests and security scans, immutable artifact, SBOM, attestation and provenance, release, verification, deployment.

Nine handoffs. The pipeline is only as strong as the weakest one, and the one most often absent is the second from the bottom.

Stage 1 — Protected source

Nothing downstream means anything if the branch being built can be modified without review.

Organisation-level rulesets so a repository admin cannot relax them:

• Require a pull request with at least one approval
• Require status checks, including security scanning results
• Block force pushes and restrict deletions
• Restrict who may create refs matching your release patterns
• Require signed commits, where you have the tooling for it

A CODEOWNERS entry for /.github/, because a workflow change is a change to what runs with your credentials.

A tag ruleset on v*, because release tags are what consumers pin to.

See Repository rulesets for security and Branch protection for security.

Stage 2 — The trusted build

The build is the highest-value target in the pipeline, because its output is what people install and it is what nobody reads.

Put the build in a reusable workflow. This creates an isolation boundary: the caller cannot influence what the build does or what the provenance claims about it. It is also what the stronger SLSA build levels require.

Pin every action to a full commit SHA. An action referenced by a moving tag is code that changes without a change on your side. See Pinning actions.

Minimise permissions per job. A build job needs contents: read plus whatever it writes. The release job needs more, and it should be a separate job.

Use OIDC rather than stored credentials. A long-lived cloud key in a repository secret works from anywhere, forever, for whoever obtains it. See Remove long-lived cloud credentials.

.github/workflows/build.yml (reusable)

on:
  workflow_call:
    outputs:
      digest:
        value: ${{ jobs.build.outputs.digest }}

permissions:
  contents: read
  packages: write
  id-token: write
  attestations: write

jobs:
  build:
    runs-on: ubuntu-latest
    outputs:
      digest: ${{ steps.build.outputs.digest }}
    steps:
      - uses: actions/checkout@v7

      - uses: docker/login-action@v4
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - id: build
        uses: docker/build-push-action@v7
        with:
          push: true
          tags: ghcr.io/${{ github.repository }}:${{ github.sha }}

      - uses: actions/attest@v4
        with:
          subject-name: ghcr.io/${{ github.repository }}
          subject-digest: ${{ steps.build.outputs.digest }}
          push-to-registry: true

The digest is an output, because everything downstream must reference the artifact by digest rather than by the tag that was pushed.

Stage 3 — Gates that actually fail

A pipeline of checks that report but do not block is a reporting pipeline.

Check	Blocks on
Tests	Failure
Code scanning	Alerts at or above a severity, via a ruleset rule
Dependency review	Vulnerable or disallowed dependencies
Secret scanning	Any finding in the diff
Container scanning	Findings above a threshold, before signing

The container scan’s position matters. Scanning after signing produces a signed vulnerable image, and downstream verifiers treat signatures as quality signals whatever the documentation says.

continue-on-error turns a gate into a log line

Every gate above has a version that reports and continues. Each is defensible individually — during adoption, for a noisy new check — and collectively they produce a pipeline that has never stopped anything.

Audit for it:

grep -rn "continue-on-error" .github/workflows/

For each result, ask what it was added for and whether that reason still holds. The answer is frequently “somebody was unblocking a release in 2024”.

Separating the jobs that hold power

A single job that builds, tests, signs and deploys holds every credential the pipeline uses, for its whole duration. Any step in it — including a third-party action three steps earlier — runs with all of them.

Splitting by privilege is one of the cheapest structural improvements available:

Job	Holds	Does
build	contents: read, registry write, id-token	Builds, pushes, attests
test	contents: read	Runs tests and scans
verify	contents: read	Checks the attestation
deploy	The environment’s credentials	Deploys a digest

The property this buys: the job that runs your test suite — which executes the most third-party code of any job in the pipeline — holds nothing but read access. A compromised test dependency cannot publish, cannot deploy and cannot sign.

It costs a little wall-clock time, because artifacts move between jobs rather than staying on one runner. That is a good trade, and passing a digest between jobs is cheap because the artifact itself lives in the registry.

Stage 4 — Build once, promote by digest

If staging and production run different binaries, everything you tested applies to a different artifact.

Build once, in the release workflow. One artifact, one digest, one set of attestations.

Promote by digest. Staging deploys image@sha256:abc…. Production deploys the same image@sha256:abc…. A tag may exist for humans; the deployment references the digest.

Verify at each promotion, so the artifact entering production is confirmed to be the one built and tested.

Rebuilding per environment produces two digests, two provenance records and two sets of test results that apply to different things. Both attestations are accurate; neither tells you the deployed artifact was tested.

Stage 5 — Evidence

Three documents, produced at build time, bound to the artifact’s digest:

Provenance. Where it came from. Generated by the platform, signed by it.

SBOM. What is inside it. Generated from the artifact, not from the repository — the two differ substantially.

Signature. That the bytes have not changed and who produced them. In practice this arrives as part of the attestation, since attestations are signed statements.

      - name: Generate the SBOM from the built image
        uses: anchore/sbom-action@v0
        with:
          image: ghcr.io/${{ github.repository }}@${{ steps.build.outputs.digest }}
          format: spdx-json
          output-file: sbom.spdx.json

      - name: Attest the SBOM against the image
        uses: actions/attest-sbom@v4
        with:
          subject-name: ghcr.io/${{ github.repository }}
          subject-digest: ${{ steps.build.outputs.digest }}
          sbom-path: sbom.spdx.json
          push-to-registry: true

Stage 6 — The release

Immutable releases lock the tag to a commit and the assets to their content, and generate a release attestation covering the tag, the commit SHA and the assets. See Immutable GitHub releases.

Draft first, then publish. Create the release as a draft, attach every asset, then publish. Once published, immutability applies — so anything missing at that point requires working around a protection that is doing its job.

Stage 7 — Approval

Production deployment should be a separate permission from pushing code.

An environment with required reviewers holds the production credentials and gates the job that uses them:

  deploy-production:
    needs: [build, verify]
    environment: production
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
    steps:
      - name: Deploy by digest
        run: ./deploy.sh "ghcr.io/${REPO}@${DIGEST}"
        env:
          REPO: ${{ github.repository }}
          DIGEST: ${{ needs.build.outputs.digest }}

The security property: secrets attached to an environment are not readable until its protection rules are satisfied. A pull request from a fork cannot reach them, and neither can a workflow somebody adds on a branch.

Stage 8 — Verification

The stage that makes every preceding one worth having, and the one most often missing.

  verify:
    needs: build
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write
    steps:
      - name: Verify provenance before deployment
        run: |
          gh attestation verify "oci://ghcr.io/${REPO}@${DIGEST}" \
            --repo "${REPO}" \
            --signer-workflow "${REPO}/.github/workflows/build.yml"
        env:
          REPO: ${{ github.repository }}
          DIGEST: ${{ needs.build.outputs.digest }}
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Three properties make this a control rather than a formality.

The expectation is stated locally. The repository and workflow come from your configuration, not from the artifact’s metadata. An artifact vouching for itself proves nothing.

It names the workflow. Not just the repository — any workflow in the repository could otherwise produce a passing attestation.

It fails the job. No continue-on-error. A failed verification stops the deployment.

Stronger still is verification at admission, which covers everything reaching the platform rather than only what your pipeline sends. See Sigstore.

The threat model

Threat. Something reaches production that was not produced by this pipeline from reviewed source — because it was substituted, because the build was influenced, or because somebody deployed around the pipeline entirely.

Attack surface. The branch being built. The workflow definition. Third-party actions. The build runner. Build credentials. The registry. The deployment configuration that selects an artifact. And the pipeline’s own gates, which fail open in ordinary ways.

Impact. The highest available: an artifact of an attacker’s choosing, distributed by your own trusted channel, running with your production access.

Control. The nine stages above, with the property that each produces evidence the next checks.

Verification. Two exercises, and the second is the one that matters.

The first: trace a real deployed artifact backwards. Which digest, which build, which commit, which pull request, which reviewer. If any link requires a guess, that is where the chain is broken.

The second: try to get something through. Push an artifact to the registry by hand and attempt to deploy it. Modify a workflow on a branch and see whether it can produce a deployable artifact. Skip the verification step and see whether anything notices. A pipeline nobody has attacked is a pipeline whose gates are untested, and gates fail open more often than they fail closed.

What each stage takes on trust

The chain is easier to evaluate when the assumptions are written down, because most of them are implicit.

Stage	Assumes	Broken by
Build	The workflow file is what maintainers intended	An unreviewed workflow change
Build	Third-party actions are what they were when reviewed	A moving tag reference
Build	The runner is clean	A persistent self-hosted runner
Artifact	The stored bytes are the built bytes	Registry write access, a moved tag
Attestation	Only the platform can produce provenance	A build step with access to signing material
Release	The published assets stay published	A mutable release
Deployment	The selected artifact is the intended one	Deploying by tag
Verification	The policy asserts the right things	An over-broad or empty constraint

The last row is worth dwelling on, because it is the one under your direct control and the one that fails most quietly. A verification step whose expected-workflow variable is empty asserts nothing and still passes. Nobody notices, because the pipeline goes green.

Adopting this incrementally

Nobody builds this in one pass, and attempting to produces a stalled project. The order below is by value per unit of disruption.

1. Deploy by digest. No new tooling, no cryptography, and it closes the tag-repointing path immediately. Changes deployment manifests only.

2. Build once and promote. Stop rebuilding per environment. This is usually a pipeline restructure rather than new tools, and it makes everything downstream coherent.

3. Remove long-lived credentials from the build. OIDC. High value, contained change.

4. Make one gate actually block. Pick the check you trust most and remove its continue-on-error. One is enough to establish that gates can fail.

5. Add provenance generation. Two lines plus permissions.

6. Add a verification step before deployment, and break it deliberately to confirm it stops things.

7. Move production credentials into an environment with required reviewers.

8. Add the SBOM and attest it.

9. Make releases immutable.

10. Move verification to admission, where the platform supports it.

Steps 1 to 3 are worth doing regardless of any supply-chain programme; they make the pipeline better in ordinary ways. Step 6 is where the security benefit starts. Steps 8 to 10 are the mature end and are worth less than 1 to 6 if those are not in place.

Break-glass, without undermining the pipeline

Every gate needs a documented way past it, or somebody will invent one under pressure and nobody will record what happened.

A break-glass path that does not hollow out the pipeline:

It is written down before it is needed. An undocumented bypass is discovered at 03:00 by someone guessing, which is the worst possible circumstance for improvisation.

It is a different path, not a disabled gate. A separate workflow, a separate approval, a break-glass environment — not commenting out the verification step, which then stays commented out.

Using it is conspicuous. An alert, a message in a channel, an issue created automatically. Something a human sees at the time.

It requires a reason. A sentence somebody will read, not a dropdown.

It creates follow-up work. A retrospective, or at minimum a note explaining what was deployed and why it could not go through the normal path.

It is rare. Weekly use means the pipeline is miscalibrated. Fix the pipeline rather than normalising the exception.

A skipped gate must not look like a passed gate

The common implementation is a label or a commit-message token that makes the verification job exit zero. That produces a green check for a check that did not run, indistinguishable in the pull request view from one that ran and passed.

Make the bypassed path report distinctly — a different check name, a neutral outcome, a comment. Somebody reading the deployment record in six months should be able to tell that verification was skipped.

Artifacts that are not containers

The examples above use container images because they are the common case. The pattern is the same for everything else, with different mechanics.

Language packages. Publish with provenance where the ecosystem supports it, and pin by version plus integrity hash on the consuming side. The registry is the equivalent of the container registry and its publish credentials deserve the same treatment.

Binaries and archives. Attest by file path with subject-path, attach to an immutable release, and publish the expected identity so consumers can verify. Checksums alongside a download are a weaker version of the same idea — useful against corruption, useless against an attacker who can replace both files.

Infrastructure. A Terraform plan applied to production is an artifact with the same questions: which commit produced it, which run applied it, against which state. Provenance for the plan is unusual and the questions are not. See Deploy Terraform.

Static sites and configuration bundles. Anything a pipeline produces and something else consumes. Cosign will sign any blob, so the pattern applies even where a purpose-built action does not exist.

The common structure across all four: produce the artifact once, identify it by content, record how it was made, and check that record at the point of use. The tooling differs; the shape does not.

Rollback

Rollback in a secure pipeline has one property that distinguishes it from an ordinary one: the previous artifact is still verifiable.

Because deployment is by digest and every artifact carries attestations, rolling back means deploying a previous digest — which is a known, tested, attested artifact rather than a rebuild.

Three things to get right:

Do not rebuild to roll back. A rebuild of an old commit produces a new artifact that was never tested and whose provenance describes a build that happened during an incident.

Verify on the way back. The same gate applies. A rollback under pressure is exactly when somebody would skip it.

Keep the artifacts. A registry retention policy that removes old images removes your rollback targets. Retention is a security decision, not just a storage one.

Infrastructure rollback is not application rollback

For an application, rolling back means running a previous artifact. The previous artifact still exists and behaves as it did.

For infrastructure, “rolling back” means applying a previous description, which is a new change moving the world toward an earlier state. Re-applying does not undo a destroy: a deleted database is gone, a released address may belong to someone else, and a recreated resource is a different resource with the same name.

Plan infrastructure recovery as forward action with explicit data-protection controls, not as an undo. See Deploy Terraform.

Putting it together

.github/workflows/release.yml

name: release

on:
  push:
    tags: ["v*"]

permissions:
  contents: read

jobs:
  build:
    uses: ./.github/workflows/build.yml
    permissions:
      contents: read
      packages: write
      id-token: write
      attestations: write

  verify:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - name: Verify the artifact's provenance
        run: |
          gh attestation verify "oci://ghcr.io/${REPO}@${DIGEST}" \
            --repo "${REPO}" \
            --signer-workflow "${REPO}/.github/workflows/build.yml"
        env:
          REPO: ${{ github.repository }}
          DIGEST: ${{ needs.build.outputs.digest }}
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

  deploy-staging:
    needs: [build, verify]
    environment: staging
    runs-on: ubuntu-latest
    steps:
      - run: ./deploy.sh staging "ghcr.io/${{ github.repository }}@${{ needs.build.outputs.digest }}"

  deploy-production:
    needs: [build, verify, deploy-staging]
    environment: production
    runs-on: ubuntu-latest
    steps:
      - run: ./deploy.sh production "ghcr.io/${{ github.repository }}@${{ needs.build.outputs.digest }}"

Read the needs chains. Production depends on staging, which depends on verification, which depends on the build. Every environment deploys the same digest. Nothing is rebuilt.

Operating this across an organisation

One well-built pipeline is a demonstration. Fifty is a different problem, and copying a workflow file fifty times produces fifty pipelines that have diverged in the direction of fewer gates.

Put the pipeline in reusable workflows. Build, verify and deploy each defined once and called by every repository. A repository’s own workflow file becomes a few lines naming which reusable workflows it uses and with what inputs.

This is also what the stronger SLSA build levels require, so it is work that counts twice.

Define the verification policy centrally. Which repositories and workflows are acceptable builders belongs in one place. A policy copied into fifty deployment workflows is fifty policies, and the divergence will not be in the strict direction.

Require the checks through organisation rulesets, targeting repositories by custom property. That is what stops a repository opting out quietly — see Repository rulesets for security.

Measure the right things. Useful numbers, in order of how much they tell you:

• How many deployments verified an attestation? Coverage of the control that matters.
• How many verifications have ever failed? A control that has never said no in six months is either a very clean pipeline or a broken check, and telling them apart requires deliberately breaking it.
• How many continue-on-error occurrences exist across your workflows? A direct count of gates that cannot block.
• How many deployments referenced a tag rather than a digest? The single best proxy for whether the chain holds end to end.

None of those is “how many repositories generate attestations”, which is the number people report and the one that says least.

Assessing your own pipeline

Ten questions. Each maps to a stage above, and the honest answers usually cluster in the same places.

1. Can anyone push to the branch you build from without review?
2. Is a workflow change reviewed by a named owner?
3. Are all your actions pinned to full commit SHAs?
4. Does the build hold any long-lived credential?
5. Does any security check report without blocking?
6. Do staging and production run the same artifact, by digest?
7. Does every artifact carry provenance?
8. Does anything verify that provenance before deployment, and fail on mismatch?
9. Is production deployment a separate approval from merging?
10. Can you roll back by deploying a previous digest without rebuilding?

Most pipelines answer badly on 5, 6 and 8. Those three are also the cheapest to fix.

What this pipeline still cannot tell you

Worth stating, because a pipeline this thorough invites more confidence than it earns.

That the code is correct. Every gate is about process and provenance. Tests tell you what the tests cover; scanning tells you what the queries model. Neither is a proof of correctness, and neither substitutes for review.

That the reviewed change was a good idea. Approval is a human judgement made under time pressure, usually by somebody who did not write the code and may not know the system it touches well.

That the dependencies are trustworthy. Pinning gives you the same bytes as last time. Whether those bytes were ever safe is dependency security’s question, and the pipeline inherits its answer.

That the deployment is configured safely. An impeccably built, verified artifact deployed with an over-permissive service account, an open network policy, or a debug flag enabled is a compromised system produced by a perfect pipeline.

That the build platform is uncompromised. The whole chain rests on GitHub’s statements about what ran. That trust is placed in an auditable system rather than an unstated assumption, which is an improvement, and it is not the same as no trust.

The point of listing these is not pessimism. It is that a pipeline like this closes a specific, well-understood set of paths — substitution, tampering, unreviewed change, credential theft — and leaves the rest to other controls. Knowing which is which is the difference between defence in depth and a false sense of coverage.

Common mistakes

Checks that report rather than block. A pipeline that has never stopped anything.

Rebuilding per environment. Your testing applies to a different artifact.

Deploying by tag. The tag can move between the build and the deployment.

Signing before scanning. A signed vulnerable image, which downstream reads as endorsed.

No verification step. Every attestation upstream is inert.

Verification that warns. Documentation.

Production credentials in repository secrets. Any workflow can read them. Environment secrets with required reviewers exist for exactly this.

Rebuilding to roll back. Deploying an untested artifact during an incident.

Retention shorter than the rollback window. The rollback target no longer exists.

Mental model

A secure release pipeline is a chain of custody where every handoff produces evidence and the next stage checks it. The evidence is worthless without the check — so the deployment gate is not the last step, it is the step that makes all the earlier ones mean something.

What you learned

• Each stage should produce evidence the next stage can verify without trusting it
• Protected source and reviewed workflow changes are the foundation everything downstream assumes
• A reusable build workflow creates the isolation the stronger SLSA build levels require
• Gates must fail; continue-on-error on a security check makes it a log line
• Scan before signing, or a signature can endorse a vulnerable artifact
• Build once and promote by digest, so what you tested is what you deploy
• Provenance, SBOM and signature are three documents bound to the same digest
• Environment protection rules keep production credentials unreachable until approval
• Verification must state the expected repository and workflow locally, and must fail the job
• Rollback means deploying a previous verified digest, never rebuilding

Exercise

Assess a real pipeline, then improve one thing.

1. Answer the ten questions above for a pipeline you own. Write the answers down.

2. Run the continue-on-error grep. Predict: how many results, and can you justify each?

3. Check whether staging and production run the same digest. Predict: does your deployment reference a tag or a digest?

4. Find the most recently deployed artifact and try to establish which commit built it. Time it.

5. Add provenance generation if it is missing. Verify it manually.

6. Add a verification step before deployment. Then break it deliberately — verify against a different workflow — and confirm the deployment stops.

7. Attempt a rollback by deploying a previous digest. Predict: does the artifact still exist in your registry?

8. Fix whichever of questions 5, 6 or 8 you answered worst.

Related lessons

Immutable GitHub releasesLocking the release at the end of this pipeline.

Artifact attestationsGenerating and verifying the evidence.

Deployment approvalsThe approval gate, in full.

Signing container imagesThe container specifics and admission policy.