Optimising CI Checkouts for Large Repositories

Checkout is the step nobody looks at, running on every job, in every pipeline, thousands of times a day.

In a large organisation with a large repository, it is frequently the single largest line item in CI spend — and unlike the build itself, almost all of it is waste. The job needs one commit’s worth of one directory. It is being given the entire history of everything.

Before you start

• Partial clone at scale
• Sparse checkout at scale

The short answer

Fetch the least the job actually needs, and know what “needs” means for that job. Most jobs need one commit of a subset of paths. Some need history, and giving them a truncated one produces wrong answers rather than errors.

The three levers, in order of typical impact:

1. Depth — fetch-depth: 1 omits history entirely
2. Filter — filter: blob:none omits file content until required
3. Sparse cone — write only the paths the job builds

They compose, and for a narrow job in a large monorepo the combination can reduce a multi-minute checkout to seconds.

The risk is always the same: a job that silently needs something you removed. Not an error — a wrong answer.

What a job actually needs

Categorise before optimising. Most pipelines contain all four kinds.

Build and unit test. One commit, one subset of paths. Aggressive optimisation is safe.

Lint and format check. One commit, and usually only the changed files. The most aggressive case.

Anything comparing against a base branch. Diff-based linting, changed-file detection, coverage comparison. Needs the merge base, and a depth-1 fetch does not have it. This is the most common breakage.

Anything reading history. Version calculation from tags, changelog generation, git describe, commit-count builds, release tooling. Needs enough history to find what it looks for, which is frequently more than people assume.

Caution

The dangerous category is the third. A tool that computes a diff against a base it cannot find does not always fail — some fall back to “compare against the empty tree”, which reports every file as changed. The lint job passes, the coverage report is meaningless, and nobody notices for months.

Depth

The largest single win for jobs that do not need history.

- uses: actions/checkout@v7
  with:
    fetch-depth: 1

fetch-depth: 1 is the default for actions/checkout. Many organisations override it to 0 — full history — once, for one job that needed it, and then copy that job as a template forever. Auditing for fetch-depth: 0 is the cheapest optimisation available, because a large fraction of the instances are cargo-culted.

When history is genuinely needed, prefer a bounded depth over unbounded:

- uses: actions/checkout@v7
  with:
    fetch-depth: 50

A bounded depth is a bet, and the bet fails silently when a pull request has more commits than the depth. If the tool needs a merge base, fetch it explicitly instead:

- uses: actions/checkout@v7
  with:
    fetch-depth: 0

# or, keep the checkout shallow and deepen only what is needed
git fetch --no-tags --depth=1 origin "$BASE_REF"
git merge-base HEAD "origin/$BASE_REF"

Tags are a separate cost. A repository with tens of thousands of tags pays for them on every fetch. fetch-tags: false and --no-tags matter more than people expect on release-heavy repositories.

Filters

- uses: actions/checkout@v7
  with:
    filter: blob:none

filter: blob:none with full history is the right default for jobs that read history. The commit graph arrives; file content is fetched only for what is checked out. A version-calculation job that walks tags needs no file content at all.

This is the combination most people miss. They assume needing history means paying for everything, so they set fetch-depth: 0 and accept a slow checkout. fetch-depth: 0 plus filter: blob:none gives complete history at a fraction of the transfer.

Note

actions/checkout exposes fetch-depth, fetch-tags, filter, sparse-checkout and sparse-checkout-cone-mode inputs. Input names and defaults have changed across major versions of the action — check the documentation for the version you have pinned rather than copying a snippet from an older pipeline.

Sparse cones in CI

CI is the ideal case for sparse checkout, because the job knows what it builds.

- uses: actions/checkout@v7
  with:
    filter: blob:none
    sparse-checkout: |
      apps/web
      libs/core
      libs/ui

Cone mode is the default for the action’s sparse-checkout input, and it should stay that way.

Include the build’s dependencies, not just its source. Root configuration files, shared tooling directories, the lockfile. The action materialises root-level files automatically under cone mode, which covers most of the common cases.

Derive the cone from the build graph if your tooling can. A monorepo build tool that knows what a target depends on can emit the cone, and a generated cone cannot drift.

Validate it. A cone missing a dependency produces a build error, which is at least loud. A cone missing a test fixture produces a skipped test, which is not.

Combining the three

The full form for a narrow job in a large monorepo:

- uses: actions/checkout@v7
  with:
    fetch-depth: 1
    fetch-tags: false
    filter: blob:none
    sparse-checkout: |
      apps/web
      libs/core

Each lever removes a different thing. Depth removes commits, the filter removes content, the cone removes paths. Removing all three leaves the job with one commit’s worth of two directories, which is what it needed.

The scripted equivalent, for systems without an equivalent action:

git clone --depth=1 --no-tags --filter=blob:none --no-checkout \
  "https://github.com/example-org/monorepo.git" repo
cd repo
git sparse-checkout init --cone --sparse-index
git sparse-checkout set --cone apps/web libs/core
git checkout "$GITHUB_SHA"

--no-checkout before configuring the cone is essential. Without it the full tree is materialised first and the saving evaporates.

The merge-base problem in detail

This deserves its own treatment because it is where most CI checkout optimisation goes wrong.

A pull request job frequently wants to know what changed. Changed-file linting, path-based job filtering, coverage deltas, dependency-review tooling. All of them compute a diff, and a diff needs two endpoints.

The correct base is the merge base, not the tip of the target branch. origin/main has moved since the branch was created; diffing against its tip reports every change anybody else made as if the pull request made it.

git merge-base HEAD "origin/$BASE_REF"

A depth-1 checkout has neither the base branch nor the merge base. The command fails, and what happens next depends entirely on the tool.

The three failure shapes, in increasing order of damage:

• The tool errors. Best case. You find out immediately.
• The tool falls back to the target branch tip. The diff is too large; the job is slow and noisy but not wrong in a dangerous direction.
• The tool falls back to the empty tree. Every file reports as added. A changed-files linter now lints everything (slow but harmless); a policy check that only inspects changed files now inspects everything and may block every pull request; a coverage-delta check reports meaningless numbers that people learn to ignore.

The fix, when you need the base and want to stay shallow:

git fetch --no-tags --depth=1 origin "$BASE_REF"
git fetch --deepen=50 origin "$BASE_REF" HEAD
git merge-base HEAD FETCH_HEAD

Or simply use fetch-depth: 0 with filter: blob:none. Complete history, no file content, and no bet about how many commits deep the merge base is. For most organisations this is the right answer, and the deepening dance is only worth it at extreme repository sizes.

Tip

Add an explicit assertion. A single line — verify the merge base resolves, and fail the job loudly if it does not — converts every one of these silent failures into an obvious one. It costs nothing and it is the highest-value line in a pull-request workflow.

Caching the repository

When the levers are not enough, cache the repository between runs.

Self-hosted runners with persistent disk can keep a clone and fetch incrementally. A git fetch on an existing clone transfers only new objects, which for a busy repository is orders of magnitude less than a clone. This is the largest available win, and it requires runners you control — see runner groups.

A local mirror or cache server near the runners. The clone is still full but the transfer is on a fast local link. Useful when runners are ephemeral but co-located.

Caching .git as a CI cache artifact works and is fiddly. The cache must be keyed so it is reused across runs, restored before checkout, and fetched to update. The saving depends on how much the cache-restore step costs relative to the clone it replaces, which for a large repository is favourable and for a small one is not.

Caution

A cached or reused clone is state that persists between jobs. Ensure the checkout resets cleanly — untracked files, modified files and stale branches from a previous job are a correctness and a security concern. git clean -ffdx and a hard reset to the target SHA, every time, before the build reads anything.

Not checking out at all

The optimisation people skip past, and frequently the largest one available.

A job that does not need to run does not need to check out anything. In a monorepo where a pull request touches one service, running every service’s build is the actual waste, and no amount of checkout tuning addresses it.

Path filters are the crude version and they are effective. A workflow that triggers only on paths: ['apps/web/**', 'libs/core/**'] does not start for a documentation change.

They are crude because they do not know the dependency graph. A change to libs/core should trigger apps/web if web depends on core, and a hand-maintained path list drifts from reality the moment somebody adds a dependency.

The build-graph version — asking the build tool which targets are affected by a diff — is correct and requires a build system that models dependencies. That is a monorepo tooling decision rather than a Git one, and it is covered in monorepo tooling at scale.

Order the work correctly: decide which jobs need to run, then make the ones that do run check out efficiently. Optimising the checkout of a job that should not have started is fixing the wrong problem.

One caution. Skipping jobs based on paths interacts with required status checks. If a check is required and the job that produces it is skipped, the pull request can block forever unless the skip is handled deliberately. Merge queues and required-check configuration both matter here; get the interaction right before rolling path filters out widely.

Ephemeral versus persistent runners

The single biggest structural factor in checkout cost, and it is a runner decision rather than a Git one.

Ephemeral runners start with nothing. Every job pays a full clone. This is the clean, secure, reproducible default, and it is why checkout optimisation matters so much on hosted runners.

Persistent runners can keep a warm clone. The job fetches the delta since the last run, which for a repository with steady commit traffic is a few hundred kilobytes rather than several gigabytes. The improvement is not incremental — it is a different order of magnitude.

The trade is isolation. A persistent working directory carries state between jobs, potentially between teams, potentially between a trusted job and an untrusted one. That is a security boundary question, and it is the reason many organisations accept the cost of ephemeral runners deliberately.

The middle ground that works: ephemeral job environments with a shared, read-only local Git cache or mirror. Each job is isolated, and the clone comes from a fast local source rather than across the internet. It requires infrastructure, and for a large organisation with a large repository it repays quickly.

Whichever you choose, choose it explicitly. The most expensive version is ephemeral runners with unoptimised full-history checkouts, which is what most organisations have by default and nobody decided.

What to measure

Checkout step duration, aggregated across all jobs. Your CI system already records this per step, and the aggregate is the number that justifies the work.

Checkout as a percentage of total job duration. If it is 40% for a two-minute job, that is the finding.

The count of jobs using fetch-depth: 0. A simple grep across workflow files, and usually a shocking number.

Cache hit rate, if you are caching. A cache that misses is slower than no cache.

Failures after a change. The metric that says whether you removed something a job needed.

# Find full-history checkouts across a fleet of workflow files
grep -rn "fetch-depth: 0" .github/workflows/

Run that across every repository and you have a prioritised worklist. For an organisation-wide sweep, the same query through the code search API turns it into a fleet operation — see API automation.

LFS in CI

A repository using Git LFS adds a step people forget to include in the optimisation.

LFS content is fetched separately from the Git objects, by the LFS client, and by default the checkout action does not fetch it at all. A job that needs LFS files must ask for them explicitly.

A job that does not need them should not fetch them. This is frequently a large saving on media-heavy repositories, and it is the LFS equivalent of a sparse cone — most jobs need the source and none of the assets.

Bandwidth is metered. LFS bandwidth consumption is billed, and a pipeline that fetches every asset on every job of every pull request can consume a monthly allowance quickly. Unlike Git object transfer, this one has a line on a bill.

Fetch selectively where the client supports it. LFS include and exclude patterns let a job pull only the assets it uses, which is the difference between a two-gigabyte fetch and a twenty-megabyte one.

Check whether the repository even still needs LFS in CI. Many jobs — linting, type checking, unit tests — never open a media file. Auditing which jobs actually read LFS content is usually quick and usually productive. The LFS article covers the storage and bandwidth model.

Rolling changes out across many pipelines

The optimisation is easy. Applying it to four hundred repositories is the actual problem.

Centralise the checkout. If teams call a reusable workflow or a composite action rather than writing their own checkout step, the optimisation is one change in one place. This is the strongest argument for standardised CI configuration, and it pays for itself the first time you change a default.

Otherwise, change defaults and let teams opt out. Publish the recommended block, offer to open the pull requests, and make the fast path the documented one.

Change one lever at a time. If a job breaks after you changed depth, filter and cone together, you learn nothing about which one did it.

Start with the jobs that run most. A five-second saving on a job that runs four thousand times a day beats a two-minute saving on a nightly.

Expect a small number of genuine breakages and treat them as information: each one is a job whose history requirement was undocumented. Write it down when you fix it.

Checkout and credentials

Optimising checkout usually means touching how the clone authenticates, and that is worth getting right rather than inheriting.

actions/checkout persists a credential in the local Git config by default so subsequent Git commands in the job can talk to the remote. That is convenient and it means a token is present in the workspace for the duration of the job.

If the job does not need to run Git commands after checkout, persist-credentials: false removes it. Most build jobs do not.

A job that pushes — a release job, a generated-file commit — does need credentials, and the token it uses should be scoped to what it pushes rather than inheriting broad permissions.

Cross-repository checkouts need explicit credentials. A job checking out a second repository cannot use the default token, and the answer is a scoped token or a GitHub App installation token rather than a long-lived personal access token stored as a secret. See machine identities.

When scripting the clone yourself, do not put a token in the URL. It ends up in process listings, in .git/config, and in any log that echoes the command. Use a credential helper or an Authorization header configured out of band.

Danger

Never embed a real token in a workflow file, a script, or documentation — use a secret reference or a placeholder. A token committed to a repository is exposed to everyone with read access and remains in history after removal. If one is committed, revoke it first and clean history second.

Common mistakes

fetch-depth: 0 copied from a template. The most common single waste.

Assuming a shallow fetch has the merge base. It does not, and the tool may not tell you.

Fetching tags on a release-heavy repository. Substantial and invisible.

Setting a sparse cone that omits a test fixture. Tests skip rather than fail.

Forgetting --no-checkout when scripting. The full tree is materialised first.

Not resetting a reused clone. Stale state across jobs.

Optimising the nightly job instead of the per-commit job. Volume is where the money is.

Changing every lever at once. Nothing is diagnosable.

Leaving the checkout step uncentralised. Every future improvement becomes a four-hundred-repository migration.

A worked prioritisation

Given a fleet and a finite amount of attention, this is the order that produces the most saving per unit of effort.

1. Count. Total checkout seconds per day across the organisation, from your CI system’s own data. Without this number nothing else can be prioritised.

2. Find the fetch-depth: 0 instances. Grep every workflow file. This alone frequently accounts for the majority of the waste.

3. Split them into “needs history” and “unknown”. The unknowns become depth-1 immediately.

4. Add filter: blob:none to everything that genuinely needs history. No behaviour change, substantial saving, essentially no risk.

5. Turn off tag fetching where tags are not read. Check release-heavy repositories first.

6. Add sparse cones to the highest-volume jobs only. The cone is the highest-maintenance lever, so spend it where the volume justifies it.

7. Then, and only then, look at caching and runner architecture. These are infrastructure projects, and steps 2 through 5 are configuration changes that may make them unnecessary.

The reason for that ordering: steps 2 through 5 are reversible one-line changes with immediate measurable effect. Steps 6 and 7 carry ongoing maintenance. Do the free things first, measure again, and see whether the expensive things are still worth doing.

Mental model

A CI checkout should be sized to the job, not to the repository. Depth, filter and cone remove three different things, and each one is safe exactly when the job genuinely does not use what it removes. The risk is never a slow build — it is a job that succeeds with less data than it needed.

What you learned

• Checkout is frequently the largest avoidable cost in CI for large repositories
• Jobs fall into four categories, and only the base-comparison and history-reading ones need care
• fetch-depth: 0 copied from templates is the most common single waste; auditing for it is the cheapest win
• fetch-depth: 0 combined with filter: blob:none gives full history at a fraction of the transfer
• Tags are a separate and substantial cost on release-heavy repositories
• Cone definitions in CI should be derived from the build graph where possible
• --no-checkout before setting the cone is what makes the scripted form actually save anything
• Reused or cached clones must be reset hard before the build reads anything
• Centralising the checkout step is what makes any of this maintainable across hundreds of repositories

Exercise

Use a repository and CI system you control, or reason through it on paper.

1. Pick one active repository. Find every workflow file and count occurrences of fetch-depth: 0.

2. For each one, determine why it is there. Record: needs merge base, needs tags, needs full history, or unknown.

3. For every “unknown”, change it to fetch-depth: 1 on a branch and run the pipeline. Predict: which fail?

4. For the ones needing history, add filter: blob:none alongside fetch-depth: 0. Compare checkout durations.

5. Pick the job that runs most often. Add a sparse cone covering only what it builds. Predict: does it still pass, and did you have to add a path you forgot?

6. Measure checkout duration as a percentage of total job time, before and after.

7. Write down, for each job you changed, what it needs from Git. That document is what prevents the next person undoing the work.

Related lessons

Partial Clone at ScaleWhat the filter is doing.

Sparse Checkout at ScaleDesigning the cones.

Standardised CI/CDMaking one change reach every repository.

Runner GroupsPersistent runners and repository caches.