Sparse Checkout at Organisational Scale

Sparse checkout is a single command for one developer and an organisational programme for a thousand.

The command is easy. What is hard is deciding who gets which paths, keeping those definitions correct as the repository moves, making CI match, and handling the day somebody’s build fails because a dependency they did not know they had is not on disk.

Before you start

• Git monorepos at scale — the repositories this applies to
• Sparse checkout — the single-repository mechanism

Caution

Git’s own documentation marks git sparse-checkout as experimental and warns that its behaviour and command-line interface may change. Non-cone mode is deprecated. The sparse index is likewise described as experimental. Verify against git help sparse-checkout for the Git version your organisation runs before standardising on it.

The short answer

Sparse checkout controls which files Git writes to your working tree. The full history and all objects are still fetched — this is a working-tree optimisation, not a transfer one. Combine it with partial clone to reduce what is downloaded as well.

At scale, use cone mode exclusively. Cone mode restricts patterns to directory prefixes, which lets Git match paths with a hash lookup instead of evaluating every pattern against every path. Non-cone mode is deprecated and its performance degrades with the number of patterns — exactly the wrong direction for a large repository.

The organisational work is the definitions, not the commands.

Cone mode and why it is the only option at scale

Cone mode accepts directories, not arbitrary globs.

git sparse-checkout set --cone apps/web libs/core
git sparse-checkout list

apps/web
libs/core

The generated pattern file shows what cone mode actually does — it materialises each parent directory’s files and then descends only into the chosen paths:

/*
!/*/
/apps/
!/apps/*/
/apps/web/

Files at the root are present. Directories at the root are excluded except the ones on the path to a selected directory. That is the cone: a set of directory cones from the root, each fully materialised at its tip.

Why this matters for performance: with directory-prefix patterns, Git can decide whether a path is included by looking up its parent directory. With arbitrary globs, it must test every pattern against every path, and a repository with 500,000 files and 200 patterns turns that into 100 million comparisons on every status.

git ls-files -t shows what the index holds — S for skipped, H for present:

S apps/api/f.txt
H apps/web/f.txt
S docs/f.txt
S libs/core/f.txt

The sparse index

Cone mode enables a further optimisation: instead of an index entry per skipped file, Git stores a single entry per skipped directory.

git sparse-checkout init --cone --sparse-index

With it enabled, git ls-files --sparse shows directory entries where the individual files used to be:

S apps/api/
H apps/web/f.txt
S docs/
H libs/core/f.txt

Why this is the difference between usable and not. Without the sparse index, a repository with 2 million files has 2 million index entries whether or not you have checked them out, and every git status reads and writes that index. With it, a developer working in a 5,000-file cone has an index proportional to their cone.

The effect is on index-reading commands — status, add, commit, checkout. These are the commands developers run constantly, so the improvement is felt continuously rather than at clone time.

Verify it is on:

git config --get index.sparse

true

Caution

The sparse index is marked experimental. Commands that have not been taught about it fall back to expanding the index to its full size, which is correct but loses the benefit — and the expansion itself is slow. If a particular command is unexpectedly slow in a sparse repository, that is usually why. This surface has improved substantially across Git versions, which is a strong argument for standardising on a recent Git across the organisation.

Designing the definitions

The technical work is one command. The organisational work is deciding what each team’s cone contains, and this is where rollouts fail.

Start from what teams already own. If the repository has CODEOWNERS, it already encodes which directories belong to which team, and that mapping is the first draft of the cone definitions.

Add the dependencies. A team’s cone must include everything their build reads, not just everything they write. Shared libraries, build configuration, tooling directories, generated protobuf definitions. This is the part people get wrong, because dependencies are invisible until a build fails.

Derive them from the build system if you can. A monorepo build tool that already knows the dependency graph can generate cone definitions. That is far more reliable than asking teams to list what they need, and it stays correct as the graph changes.

Store them in the repository. A sparse-profiles/ directory with one file per team, version-controlled and reviewable, means the definitions change through pull requests like everything else.

sparse-profiles/
  web.txt
  api.txt
  data-platform.txt
  full.txt

Provide a script that applies them:

#!/usr/bin/env bash
set -euo pipefail
profile="${1:?usage: sparse-apply <profile>}"
file="sparse-profiles/${profile}.txt"
[ -f "$file" ] || { echo "no such profile: $profile" >&2; exit 1; }
git sparse-checkout init --cone --sparse-index
xargs git sparse-checkout set --cone < "$file"

Include a full profile that disables sparse checkout. People need to grep the whole repository sometimes, and if the only way to do that is to remember the disable command, they will instead conclude sparse checkout is the problem.

Rolling it out

1. Confirm the Git version floor. Sparse index behaviour and command coverage improve materially across releases. Decide a minimum, verify what people are actually running, and upgrade before rolling out rather than debugging version-specific behaviour afterwards.

2. Make it opt-in first. Ship the profiles and the script, document them, and let interested teams try it. The people who adopt early find the missing dependencies for everyone else.

3. Fix the profiles from real failures. Every “my build broke” report is a missing path in a definition. Fix the definition, not the individual’s checkout, so the fix reaches everybody.

4. Measure. Time git status and a full build before and after for a representative developer. If the numbers are not compelling, the rollout will not survive contact with the first problem.

5. Make it the default for new clones. A setup script that clones with the right profile means new joiners never experience the full checkout.

6. Leave existing clones alone. Converting a working clone is possible and rarely worth mandating. Sparse checkout applies to new work naturally.

Do not mandate it before the profiles are right. A forced rollout with incomplete definitions generates a support queue that will end the programme.

Sparse checkout in CI

CI is where sparse checkout produces the clearest, most measurable win — and where it is easiest to get right, because the job knows exactly what it needs.

A job that builds one service needs one service’s paths. No exploration, no grep across the tree, no ambiguity. The cone is knowable from the job definition.

git clone --filter=blob:none --no-checkout --depth=1 \
  https://github.com/example-org/monorepo.git
cd monorepo
git sparse-checkout init --cone --sparse-index
git sparse-checkout set --cone apps/web libs/core
git checkout

--no-checkout matters. Without it, the clone materialises the entire working tree first and the sparse configuration only applies afterwards — you pay the full cost and then discard it.

Combine with --filter=blob:none so the objects for excluded paths are never downloaded either. Sparse checkout alone still transfers everything. See partial clone at scale for what that filter does and its failure modes.

--depth=1 on top if the job does not need history. Most build jobs do not.

Note

actions/checkout exposes sparse-checkout and sparse-checkout-cone-mode inputs, along with filter and fetch-depth. Using the action’s inputs is preferable to scripting the clone yourself, because the action handles authentication and the various fetch modes correctly. Check the action’s current documentation for the exact input names and defaults in the version you pin.

Derive the cone from the changed paths for pull request jobs, and CI cost drops further — a job that only needs to test what changed only needs to check out what changed. That is a build-graph question rather than a Git one, and it is the monorepo CI problem.

Sparse checkout and the shared history

A concern that comes up in every rollout and deserves a direct answer: sparse checkout does not change what you commit, what you push, or what anybody else sees.

The repository is identical. Every developer has the full object database (or, with partial clone, fetches on demand). The commits you create contain the full tree, because the index still knows about every path — it simply marks most of them skipped.

Merges and rebases work normally. Git materialises files it needs to resolve, resolves them, and returns to the sparse state. A conflict in a path outside your cone appears as an ordinary conflict.

A git pull that changes files outside your cone updates them in the index and not on disk. Nothing is lost; you simply do not see the change. Running git sparse-checkout disable at any point reveals the current state of everything.

You cannot accidentally delete files by narrowing your cone. Narrowing removes them from the working tree, not from the repository. This is the fear that stops adoption, and the demonstration in the exercise below is the fastest way to dispel it.

The one genuine asymmetry: if you have uncommitted changes in a path and then exclude that path, Git will refuse rather than discard them. That is correct behaviour, and the error message says so.

What goes wrong at scale

The problems that appear with a thousand developers and not with ten.

Missing transitive dependencies. A team’s cone covers what they edit but not a shared library their build imports. The failure appears as a confusing build error rather than “file not found”, because the build system reports a missing module.

Definitions drifting from the repository. A directory is renamed, and every profile referencing the old path silently stops matching it. Nothing errors — the path is simply not checked out. Add a CI check that every path in every profile exists.

Tooling that assumes the full tree. Linters, code generators, IDE indexers and scripts that walk the repository from the root. They do not fail loudly; they produce results based on a partial view. This is the most damaging category because it is silent.

Grep and search returning wrong answers. A developer searches for a symbol, finds nothing, and concludes it does not exist. It exists — it is just not on disk. Teach the full profile as the answer, and use the code host’s search for repository-wide questions.

Commands that expand the sparse index. A command without sparse index support inflates the index to its full size, which is slow, and leaves it that way until something collapses it again. Symptom: the first slow command makes every subsequent command slow.

Merge conflicts in files outside the cone. Git handles this — it materialises what it needs — but the developer sees a conflict in a path they have never opened. The resolution is normal; the confusion is not.

People disabling it and not telling anyone. Someone hits a problem, runs git sparse-checkout disable, and their clone is full again. Their subsequent feedback about performance is about a configuration they are no longer running.

The developer experience problem

Performance is why sparse checkout is adopted. Developer experience is why it is abandoned.

Editors and IDEs see a partial tree. Go-to-definition into a path outside the cone fails. Some language servers handle this gracefully by falling back to the index they can build; others report the symbol as undefined. Test this with the editors your organisation actually uses before rolling out, because the answer varies widely and it is the first thing developers notice.

Repository-wide search does not work locally. grep -r and the editor’s find-in-files search what is on disk. The honest answer is that repository-wide search belongs on the code host, and that is a genuine behaviour change people need to be told about explicitly rather than discovering.

Adding a file outside the cone is confusing. Git will not stage a path the sparse definition excludes without --sparse, and the error message is not self-explanatory to somebody who does not know sparse checkout is in play.

The mitigation is documentation, not configuration. A short page that says: here is your cone, here is how to see it, here is how to add a path, here is how to turn it off entirely, here is where to search the whole repository. Five commands. Most of the support load disappears when that page exists.

And an escape hatch that works. git sparse-checkout disable is instant, lossless and reversible. A developer who knows they can leave at any time tolerates the constraints; one who feels trapped files a complaint.

Measuring whether it worked

Decide the numbers before the rollout, because afterwards everybody will have an opinion and nobody will have data.

git status wall time, on a representative developer machine, in the same repository state. This is the number developers feel, because they run it dozens of times a day — directly and through their shell prompt and editor.

Files on disk and index size. git ls-files | wc -l and the size of .git/index. These explain the status number.

Clone-to-first-build time for a new joiner. The onboarding metric, and the one that justifies the work to management.

CI job duration for a checkout step, before and after. Usually the largest single percentage improvement, and the easiest to measure because CI already records it.

Support tickets mentioning the repository. The number that tells you whether the definitions are right. It should spike during opt-in and then fall below the baseline.

Tip

Capture the before numbers on a machine you can re-measure on later. “It feels faster” does not survive the first person who says it feels slower, and without a baseline you cannot tell whether a later regression came from sparse checkout, a Git upgrade, or the repository simply growing.

When sparse checkout is not the answer

It is a good tool for a specific shape of problem, and reaching for it elsewhere wastes effort.

A repository that is large because of history, not breadth. If the tree is 4,000 files but .git is 12 GB, the problem is object history. Sparse checkout changes nothing. Partial clone or a history cleanup is the answer.

A repository where everybody touches everything. If the cones would all be near-total, you have added configuration and support load for no reduction.

A repository that should be several repositories. Sparse checkout can make an accidental monorepo tolerable, and that is a legitimate reason to use it. But if the components have no shared build, no shared release and no shared ownership, the structural question is worth asking before the tooling one — see monorepo versus polyrepo.

A team of five. The operational cost of maintaining profiles is real, and it only pays back across many developers and many pipelines.

Verifying a checkout is what you think

Three commands worth knowing and documenting.

git sparse-checkout list          # the current cone
git config --get core.sparseCheckout
git config --get index.sparse

git sparse-checkout list is the first thing to ask for in any report of a build failure in a sparse repository. The answer is frequently the whole diagnosis.

git sparse-checkout reapply re-evaluates the current definition against the current commit. Useful after a merge that added directories, and after a Git upgrade.

git sparse-checkout disable restores the full working tree. Everything comes back; nothing is lost. Say this loudly in your documentation, because people fear the command far more than they should.

Keeping definitions correct over time

Profiles are the part that rots, and the rot is silent. Three mechanisms keep them honest.

A CI check that every profile path exists. Cheap to write, and it catches the directory-rename failure the same day it happens rather than the next time somebody applies that profile:

#!/usr/bin/env bash
set -euo pipefail
status=0
for profile in sparse-profiles/*.txt; do
  while read -r path; do
    [ -z "$path" ] && continue
    if [ ! -d "$path" ]; then
      echo "::error::$profile references missing path: $path"
      status=1
    fi
  done < "$profile"
done
exit $status

Ownership on the profile files. A CODEOWNERS entry putting sparse-profiles/web.txt under the web team means the people who feel the consequences approve the changes.

A periodic regeneration from the build graph, if your build tool can produce one. Generated definitions cannot drift from the dependency graph, because they are derived from it. This is the only mechanism that scales past a few dozen profiles, and it is worth the investment for a repository with hundreds of teams.

Treat a directory rename as a profile change. The pull request that renames libs/core to libs/platform should update every profile referencing it, and the CI check above is what makes that non-optional.

The relationship to other techniques

Sparse checkout is one of three things people confuse.

Technique	Reduces	Does not reduce
Sparse checkout	Files written to the working tree	Objects downloaded
Partial clone	Objects downloaded	Working-tree size
Shallow clone	Commits downloaded	Working-tree size

They compose, and for CI the combination of all three is usually right. For developers, sparse checkout plus partial clone with full history is the common configuration — history matters interactively, and the missing objects are fetched on demand.

Sparse checkout alone does not make cloning faster. If somebody reports that they enabled it and clone time did not change, that is the expected result, and the fix is --filter=blob:none.

Common mistakes

Using non-cone mode. Deprecated, and its performance degrades exactly where you need it.

Forgetting --sparse-index. The index is where the benefit lives for large trees.

Omitting --no-checkout when cloning. You materialise the full tree and then throw it away.

Expecting a faster clone from sparse checkout alone. It is a working-tree optimisation.

Cone definitions listing only what a team edits. Builds need dependencies.

No CI check that profile paths exist. Definitions rot silently after directory renames.

Mandating it before the definitions are correct. The support load ends the programme.

No documented full profile. People need to search the repository.

Assuming tooling handles a partial tree. Indexers and generators frequently do not, and they fail quietly.

Standardising on it with an old Git. Sparse index command coverage has improved substantially; an old client gives a worse experience for no reason.

Mental model

Sparse checkout is a per-developer view of a shared repository. The command is trivial; the view definitions are a maintained artifact that must track the repository’s structure and each team’s build dependencies. Treat them as code — version-controlled, reviewed, and CI-validated.

The failure mode is always a stale or incomplete definition, and the fix is always to the definition rather than to the individual’s machine.

What you learned

• Sparse checkout controls the working tree, not what is downloaded; combine with partial clone for transfer savings
• Cone mode restricts patterns to directory prefixes so matching is a lookup rather than a scan; non-cone is deprecated
• The sparse index stores one entry per skipped directory and is what makes very large repositories usable
• The sparse index is experimental, and commands without support for it expand the index and lose the benefit
• Cone definitions must include build dependencies, not just owned paths
• Store profiles in the repository, apply them with a script, and CI-check that their paths still exist
• In CI, --no-checkout plus --filter=blob:none plus a cone is the effective combination
• git sparse-checkout disable restores everything, and saying so reduces resistance
• The most damaging failures are silent: tooling that walks the tree and produces answers from a partial view

Exercise

Use a disposable repository.

1. Create a repository with apps/web, apps/api, libs/core and docs, each containing a file. Commit.

2. Run git sparse-checkout set --cone apps/web. List the files on disk. Predict: what is present?

3. Read .git/info/sparse-checkout. Explain each line.

4. Run git ls-files -t. Identify the S and H markers.

5. Enable the sparse index with git sparse-checkout init --cone --sparse-index, then run git ls-files -t --sparse. Predict: how does the output change?

6. Add libs/core with git sparse-checkout add. Confirm with git sparse-checkout list.

7. Run git sparse-checkout disable. Predict: what comes back, and is anything lost?

8. Write a two-team profile set for a repository you actually work on. For each team, list the paths they edit and the paths their build reads. Note which of the second list you had to look up.

Related lessons

Partial Clone at ScaleReducing what is downloaded, not just written.

CI Clone OptimisationWhere these techniques pay off most.

Git Monorepos at ScaleThe repositories that need this.

Sparse CheckoutThe single-repository mechanism.