Partial Clone at Organisational Scale

Partial clone is the technique that makes a genuinely large repository possible to clone at all.

It is also the one whose failure mode is the least intuitive, because a partially cloned repository looks complete until an operation needs something that is not there, and then it goes to the network in the middle of a command that has never touched the network before.

Before you start

• Partial clone — the single-repository mechanism
• Sparse checkout at scale — the complementary technique

The short answer

Partial clone omits objects at clone time and fetches them on demand. The repository knows the objects exist, knows their identity, and knows where to get them. When a command needs one, Git fetches it transparently.

--filter=blob:none is the filter to use. It omits every file’s content and keeps all commits and trees. History operations work at full speed; file content arrives when you check something out or diff it.

The remote that supplies the missing objects is a promisor remote. If it is unreachable, operations needing absent objects fail — this is the central operational property.

Partial clone reduces what you download. Sparse checkout reduces what is written to disk. They are different axes and they compose.

Demonstrating what it actually does

The clearest way to see partial clone is to clone without checking out, so no on-demand fetch has happened yet.

git clone --filter=blob:none --no-checkout <url> repo
cd repo
git count-objects -vH | grep size-pack

size-pack: 1.86 KiB

The objects Git knows about but does not have are listed by rev-list:

git rev-list --objects --all --missing=print | grep -c '^?'

6

Then check out, which triggers the fetch:

git checkout main
git count-objects -vH | grep size-pack

size-pack: 149.71 KiB

The second pack is the on-demand fetch. Git contacted the promisor remote, asked for exactly the blobs the checkout needed, and stored them in a new pack.

Note

The ? prefix in --missing=print output marks an object Git knows about but does not have locally. On a fully cloned repository this list is empty; on a partial clone it is the set of objects that would trigger a network round trip.

Promisor remotes

The mechanism that makes the missing objects legitimate rather than corruption.

git config --get remote.origin.promisor
git config --get remote.origin.partialclonefilter

true
blob:none

promisor=true tells Git this remote promises to supply any object it advertised. Without it, missing objects mean a corrupt repository and Git says so. With it, they mean “fetch when needed.”

Packs from a promisor remote carry a .promisor marker file alongside the usual .idx and .pack:

pack-7ea8044f….idx
pack-7ea8044f….pack
pack-7ea8044f….promisor

The filter is recorded so subsequent fetches apply the same one. A git fetch on a blob:none clone does not suddenly download every blob.

The server must permit filtering. With uploadpack.allowFilter disabled, Git warns and clones fully:

warning: filtering not recognized by server, ignoring

That warning is easy to miss in CI logs, and the symptom is a job that is mysteriously not faster. GitHub supports partial clone; a self-hosted or intermediate mirror may not.

Choosing a filter

Three filters matter in practice.

--filter=blob:none — omit all file contents. Keep all commits and trees. This is the default choice. History browsing, git log, git blame on the commit graph, branch operations: all local. Content arrives on checkout and diff.

--filter=blob:limit=<size> — omit blobs above a size. Keeps small files local and defers large ones. Useful for a repository whose problem is a handful of large files rather than breadth. --filter=blob:limit=1m is a reasonable starting point.

--filter=tree:0 — omit trees as well as blobs. The smallest possible clone. Almost never right for a developer, because any command that inspects a path needs trees, which means network traffic for operations that feel purely local. It has a place for a CI job that only needs HEAD.

Filter	Keeps	Best for
blob:none	All commits and trees	Developers, most CI
blob:limit=1m	Everything under the limit	Repositories with a few large files
tree:0	Commits only	Narrow CI jobs, mirrors

Combine with --depth only deliberately. A shallow partial clone is small, but it removes history that partial clone was specifically designed to keep cheap, and shallow clones have their own deepening cost.

Where partial clone belongs

Developer clones of large repositories. This is the primary case. --filter=blob:none with full history gives fast cloning while keeping every history operation local. Combine with a sparse checkout so only the relevant paths are written.

git clone --filter=blob:none --sparse <url>
cd repo
git sparse-checkout init --cone --sparse-index
git sparse-checkout set --cone apps/web libs/core

CI jobs. Almost always beneficial, because a job needs one commit’s content and nothing else.

Mirrors and caches. With care — a mirror that is itself partial cannot serve as a promisor for others.

Where it does not belong: anything that must work offline, anything that will run history-wide analysis, and backups.

Danger

A partial clone is not a backup. It does not contain the repository. If the promisor remote is lost, the missing objects are unrecoverable from that clone. Any backup or archival copy must be a full clone, and --mirror does not change this — a mirror of a partial clone is still partial.

The operations that go badly

The list worth knowing before rollout, because each of these is a support ticket.

Anything that walks history’s content. git log -S, git log -p over a long range, git grep on old commits. Each needs blobs Git does not have, so each fetches them — one round trip after another. A command that takes two seconds on a full clone can take minutes.

git blame on a long-lived file. Blame needs the content of every revision it walks. On a file with 800 commits, that is 800 blobs. It completes, and it is slow the first time.

Bulk git checkout across distant commits. Bisecting through history fetches a new content set at each step.

Working offline. Every absent object is a hard failure. A developer on a plane has a repository that works for anything already fetched and fails for anything else, with an error that does not obviously say “you are offline.”

Repacking and garbage collection. Objects in promisor packs are handled differently from ordinary ones. git gc copes, but housekeeping on a partial clone is more subtle than on a full one — see git maintenance at scale.

Tooling that assumes local objects. Scripts that read .git/objects directly, or that call plumbing without expecting a fetch, misbehave. This category grows with the amount of home-grown tooling an organisation has.

The one that surprises people most: these operations do not fail. They succeed, slowly, with network traffic. There is no error to search for — just an unexplained pause.

What the filters actually cost to compute

Worth knowing because it shapes server behaviour under a large rollout.

A filtered clone is not simply a smaller pack. The server must walk the object graph and decide which objects the filter admits, which is more work than sending a pack it may already have cached. On a very large repository the first filtered clone of the day can be slower to start than an unfiltered one, even though it transfers far less.

blob:none is the cheapest filter to evaluate — the decision is per-object-type and needs no size lookup.

blob:limit=<size> requires knowing each blob’s size, which is available but adds work.

tree:0 produces the smallest transfer and shifts the cost to every subsequent operation instead.

GitHub handles this transparently and there is nothing to tune. On a self-hosted instance, a rollout that turns a thousand daily full clones into a thousand daily filtered clones plus tens of thousands of small on-demand fetches is a different load profile, and it is worth telling whoever runs the instance before you do it.

Rolling it out

1. Confirm the server supports filtering. Clone with a filter and check for the filtering not recognized by server warning. Do this against every host developers actually clone from, including mirrors and proxies.

2. Start with CI. The wins are largest, the failure modes are smallest, and no human is inconvenienced while you learn.

3. Measure clone time and transfer size. git count-objects -vH before and after. These are the numbers that justify the change.

4. Offer it to developers as an opt-in clone command, documented alongside the sparse profile. New clones only — converting existing ones is possible and rarely worth it.

5. Document the offline limitation explicitly. This is the single most important line in the documentation, and the one most likely to be omitted.

6. Watch for slow-blame and slow-log reports. They are the expected cost, and the answer is either patience or a full clone for that person’s workflow.

7. Decide the default for new joiners once the failures are understood, not before.

Diagnosing a partial clone in the field

When somebody reports that something is slow or broken in a repository you did not set up, four commands establish the situation.

git config --get remote.origin.promisor         # is this a partial clone at all?
git config --get remote.origin.partialclonefilter   # which filter?
ls .git/objects/pack/*.promisor 2>/dev/null | wc -l  # how many promisor packs?
git rev-list --objects --all --missing=print | grep -c '^?'   # how much is absent?

The last number is the diagnosis. A large count means most operations touching content will go to the network. A count of zero on a repository configured as partial means everything has already been fetched, and the clone now has the same size as a full one — which happens naturally over months of use.

That drift is worth knowing about. Partial clone saves at clone time; it does not keep the repository small forever. A developer who has been working in a blob:none clone for a year has fetched most of what they touch, and their .git directory reflects it. This is fine — they still avoided the initial download of everything they never touched — but it means “our clones stay small” is not the claim to make.

Slowness with no error and periodic pauses is the signature of on-demand fetching. Running the command with GIT_TRACE=1 shows the fetch happening.

“Object not found” style errors with the remote unreachable is the offline case, and the answer is connectivity, not repository repair. Resist the instinct to run git fsck and start deleting things: on a partial clone, fsck will report missing objects that are supposed to be missing.

Cost and where it lands

Partial clone moves cost around rather than removing it, and knowing where it lands prevents surprises.

Clone time falls sharply. This is the headline and it is real.

Server load changes shape. Instead of one large pack negotiation per clone, the server handles many small fetch requests over the clone’s lifetime. For a hosted service this is somebody else’s problem; for a self-hosted instance it is a capacity question worth raising before a large rollout.

Total bytes transferred may go up or down. A developer who eventually touches most of the repository transfers more in aggregate, because on-demand fetches are less efficient than one big pack. A developer who touches a tenth of it transfers far less. Both are common.

Latency becomes a factor it was not before. On a full clone, git checkout of an old branch is local. On a partial clone it is a network round trip, and a developer on a high-latency link feels that in a way a colleague in the same region as the server does not.

The calculation that matters: partial clone is clearly worth it when the repository is much larger than any individual’s working set. It is marginal when everybody eventually touches everything.

Partial clone versus shallow clone

They are frequently confused and they solve different problems.

	Partial clone	Shallow clone
Omits	Objects (by filter)	Commits (by depth)
History	Complete	Truncated
Missing data	Fetched on demand	Requires explicit deepening
git log over full range	Works	Does not
git blame	Works, slowly	Fails past the depth
Pushing	Normal	Can be problematic
Suitable for developers	Yes	Rarely

Shallow clone is a CI tool. For a job that builds one commit, --depth=1 is correct and cheap.

Partial clone is the developer tool, because it preserves the history operations that make Git useful interactively.

They combine for CI, and the combination is usually the fastest checkout available:

git clone --depth=1 --filter=blob:none --no-checkout <url>

Caution

Shallow clones interact badly with some workflows. Pushing from a shallow clone is restricted, tools that need merge bases can fail, and deepening later is a separate fetch. For anything beyond a single-commit build job, prefer partial clone with full history.

Composing with sparse checkout

The pairing that most large-repository setups converge on.

Partial clone means the blobs for excluded paths are never downloaded. Sparse checkout alone still transfers everything and simply declines to write it.

Sparse checkout means the blobs for included paths are the only ones fetched on demand. Partial clone alone would fetch content for the whole tree on checkout.

Together, the transfer is proportional to the cone. That is the property that makes a repository with millions of files workable for a team that touches a thousand of them.

git clone --filter=blob:none --sparse <url> repo
cd repo
git sparse-checkout set --cone apps/web libs/shared

--sparse on clone initialises a cone containing only the root-level files, so nothing else is materialised before you set the real cone. Without it, the initial checkout fetches the full tree’s content and the saving is lost.

Partial clone and Git LFS

Two mechanisms that solve overlapping problems and are frequently confused.

LFS replaces file content with a pointer at commit time. The repository permanently contains pointers; the content lives on a separate LFS server. This is a change to what is committed.

Partial clone leaves the repository unchanged and defers the transfer. Nothing about the commits differs.

They are not alternatives so much as different eras. LFS predates partial clone and requires a decision at commit time, applied through .gitattributes, affecting everyone forever. Partial clone is a clone-time decision, per-person, reversible, and requires no change to the repository.

For a new repository with large-but-versioned files, partial clone plus sparse checkout increasingly does what LFS was adopted for, without the client dependency or the migration. The remaining reason to choose LFS is GitHub’s 100 MB per-object limit, which partial clone does not lift — a 500 MB file cannot be pushed at all, filtered or not.

They coexist without conflict. A repository can use LFS for a few very large assets and be cloned partially. LFS content is fetched by the LFS client on checkout; other blobs are fetched by Git’s promisor mechanism. The two paths do not interact.

A caution on migrations: if you are considering LFS specifically to reduce clone times, measure what --filter=blob:none gives you first. It is a configuration change rather than a history rewrite, and it is frequently enough. The LFS article covers the full comparison.

Setting an organisational default

Once the failure modes are understood, the question becomes what people get without thinking about it.

A clone wrapper is the effective mechanism. A script or internal CLI that clones with the organisation’s chosen filter and applies the right sparse profile. People run one command, get a correct setup, and never learn the flags.

clone.filter can be set in global config so plain git clone applies a filter, but be careful: it applies to every repository the developer clones, including small ones where it is pointless and open-source ones whose servers may not support filtering. A wrapper scoped to your own hosts is safer.

In CI, put it in the shared workflow. actions/checkout accepts a filter input, and setting it in a reusable workflow means every job inherits it without each team deciding. That is the same standardisation argument as centralised CI configuration.

Document the escape hatch. git clone with no filter always works. Somebody doing history archaeology, or working offline for a week, should know how to get a full clone without asking permission.

Revisit the default after a Git upgrade. Partial clone behaviour, and the set of commands that handle missing objects gracefully, has improved across releases. A decision made three versions ago is worth re-testing.

Common mistakes

Ignoring the filtering not recognized by server warning. The clone is full and nobody notices.

Using tree:0 for developer clones. Every path-inspecting command becomes a network operation.

Treating a partial clone as a backup. It is not a copy of the repository.

Not documenting the offline limitation. The most common source of confusion.

Expecting git blame to be fast. It fetches every revision’s content the first time.

Combining --depth=1 with partial clone for developers. You lose history for no additional benefit.

Forgetting --sparse or --no-checkout when combining with sparse checkout. The initial checkout fetches everything.

Rolling out to developers before CI. Learn the failure modes where they are cheap.

Measuring the result

The numbers to capture, and the trap in each.

Clone wall time, from a machine with representative network conditions. Trap: measure a cold clone, not one where the server has a warm pack cache from your previous attempt.

Transfer size, via git count-objects -vH immediately after clone. Trap: run it before checking out, or the on-demand fetch is included and the number looks worse than it is.

Time to first successful build. Trap: this includes the on-demand fetch for the checkout, so it is the honest end-to-end number — and it is the one that should be quoted rather than the raw clone time.

.git size after a month of normal use. Trap: people quote day-one numbers and are then surprised. Measure the steady state too.

CI checkout step duration, which your CI system already records per job. Trap: none, and this is why CI is the right place to start — the measurement is free and the population is large.

Mental model

A partial clone is a repository with a standing debt to its remote. It knows what it owes and it settles up when a command needs the object. Everything works, and some things become network operations that were not before.

The corollary: the value of partial clone is proportional to how much of the repository you never touch, and its cost is proportional to how often you reach for something you skipped.

What you learned

• Partial clone omits objects at clone time and fetches them on demand from a promisor remote
• --filter=blob:none is the default choice; it keeps all commits and trees so history operations stay local
• tree:0 is too aggressive for developers because path inspection becomes network traffic
• .promisor marker files and remote.origin.promisor=true are what make missing objects legitimate
• git rev-list --missing=print shows exactly which objects are absent
• The server must have uploadpack.allowFilter enabled, and the warning when it does not is easy to miss
• git blame and git log -S still work but become slow, with no error to diagnose
• A partial clone is not a backup and cannot be restored without its promisor remote
• Combine with sparse checkout and --sparse so transfer is proportional to the cone
• Shallow clone truncates commits and belongs in CI; partial clone preserves history and belongs on developer machines

Exercise

Use disposable repositories. You will need a source repository with uploadpack.allowFilter enabled.

1. Create a source repository with three commits, each adding a large binary file and a text file. Enable filtering: git config uploadpack.allowFilter true.

2. Clone it with --filter=blob:none --no-checkout. Run git count-objects -vH. Predict: roughly how large?

3. Run git rev-list --objects --all --missing=print | grep '^?'. Predict: how many objects are listed, and what kind?

4. Check out the default branch. Re-run git count-objects -vH. Explain the change and count the packs in .git/objects/pack/.

5. Find the .promisor files. Explain what they mark.

6. Clone the same source without uploadpack.allowFilter on the source. Read the warning. Confirm the clone is full.

7. Disconnect the source (rename the directory). Run git log --oneline — does it work? Now run git log -p — does it?

8. Restore the source. Delete both clones.

Related lessons

Sparse Checkout at ScaleThe complementary working-tree technique.

CI Clone OptimisationWhere these filters pay off most.

Git Maintenance at ScaleHousekeeping on partial clones.

Partial CloneThe single-repository mechanism.