Commit-Graph and Multi-Pack-Index at Scale

Two files decide whether a large repository feels fast, and most people have never heard of either.

The commit-graph makes history traversal cheap. The multi-pack-index makes object lookup cheap when there are many packs. Neither changes what the repository contains — they are caches, derived entirely from data Git already has, and either can be deleted and rebuilt.

Before you start

• Git objects — commits, packs and the object database
• Git maintenance at scale — what builds these on a schedule

The short answer

The commit-graph is a precomputed index of the commit history’s shape. Parent relationships, commit dates, and generation numbers, in a format Git reads directly instead of decompressing and parsing thousands of commit objects.

The multi-pack-index is a single index across many pack files. Without it, finding an object in a repository with fifty packs means consulting fifty pack indexes.

Both are enabled by default for reading — core.commitGraph and core.multiPackIndex both default to true. The question is whether anything is writing them.

The commit-graph and incremental-repack maintenance tasks write them. If maintenance is not running, these files are stale or absent and the repository is slower than it needs to be.

What the commit-graph accelerates

Every operation that walks history without needing file content.

git log with any traversal. Ordering commits requires knowing their parents and dates. Without the graph, that means reading and decompressing every commit object on the walk.

git merge-base. Finding the common ancestor of two branches is a graph problem, and it is the operation behind every merge, every rebase, and every pull request’s diff.

git branch --contains and git tag --contains. Reachability questions over the whole ref set.

git log --graph, git rev-list --count, ahead/behind calculations. Everything a status display shows about how far a branch has diverged.

The magnitude on a large repository is not subtle. A merge-base on a history with a million commits goes from seconds to milliseconds, and because merge-base is invoked by so many higher-level operations, the effect compounds.

Generation numbers

The idea that makes graph traversal efficient rather than merely cached.

A generation number is a commit’s distance from the root, precomputed and stored. Given two commits’ generation numbers, Git can frequently conclude one cannot be an ancestor of the other without walking anything.

Version 2 uses corrected commit dates, which handles repositories where commit timestamps are not monotonic — the common case, since clocks are wrong and rebases reorder things.

git config --get commitGraph.generationVersion

commitGraph.generationVersion defaults to 2. Version 1 omits the corrected commit dates. There is no reason to set it to 1 except compatibility with a very old Git reading the same object directory.

Why corrected dates matter: a naive traversal that stops when it reaches commits older than the target relies on timestamps being ordered. When they are not, the traversal either stops too early — a wrong answer — or cannot use the optimisation at all. The corrected date is a value Git computes to be consistent with the graph regardless of what the recorded timestamp says.

Changed-path Bloom filters

The optional extra that transforms path-limited history.

git commit-graph write --reachable --changed-paths

This computes and stores information about the paths changed between a commit and its first parent. The manual states directly that it provides significant performance gains for git log -- <path>, and warns the operation can take a while on large repositories.

What it does at query time: git log -- some/deep/directory/file.c normally has to diff every commit against its parent to decide whether that path changed. With Bloom filters, most commits are eliminated by a cheap probabilistic check, and only candidates are diffed.

A Bloom filter can produce false positives but never false negatives, so the answer stays correct — some commits are diffed unnecessarily, none are wrongly skipped.

The setting is sticky. The manual notes that once given, future commit-graph writes assume it was intended; --no-changed-paths stops storing the data.

--max-new-filters=<n> bounds the work per write, with -1 meaning no limit. Only commits in the new layer count against it. The manual advises --split=replace to retroactively compute filters over earlier layers.

Tip

On a large monorepo, changed-path Bloom filters are the single highest-value optimisation for developers, because git log on a path is how people investigate anything. Building them the first time is slow; use --max-new-filters to spread the cost across several maintenance runs rather than one very long one.

Split commit-graphs

Writing a whole commit-graph on every update would be prohibitive on a large repository. The split form solves that.

git commit-graph write --reachable --split
ls .git/objects/info/

commit-graph-chain
graph-60af76ddbc0ac153634af85e34080a4fdb29317d.graph

The graph becomes a chain of files in <dir>/info/commit-graphs, with commit-graph-chain listing them in order. New commits go into a new small layer; layers are merged over time according to a strategy.

This is what makes hourly maintenance viable. Adding an hour’s commits writes a tiny file rather than rewriting a structure covering a million commits.

The maintenance task uses the split form, and the manual notes the incremental write is safe alongside concurrent Git processes because it does not expire .graph files referenced by the previous chain — those are removed by a later run based on the expiry delay.

Note the two locations. A single-file graph is .git/objects/info/commit-graph; a chain lives in .git/objects/info/commit-graphs/ with the chain file alongside. Both may be present during a transition.

Writing and verifying

git commit-graph write --reachable
git commit-graph verify

--reachable walks from all refs. The other input modes exist for specific purposes: --stdin-packs walks only the named pack indexes, --stdin-commits takes a list of commit IDs.

verify reads the file and checks it against the object database. It is the command for “is this file corrupt”, and it exits non-zero when it is not. --shallow checks only the tip file of a chain.

If core.commitGraph is disabled, write warns and returns success without writing. A silent no-op that is worth knowing about when a script appears to work and produces nothing.

fetch.writeCommitGraph writes a commit-graph after every fetch that downloads a pack. Reasonable for a repository not covered by scheduled maintenance; redundant when it is.

The multi-pack-index

The second structure, solving a different problem.

A repository accumulates pack files. Each fetch may add one. Each maintenance run may add one. Without consolidation, a busy repository ends up with dozens.

Every object lookup consults every pack index. With fifty packs, that is fifty binary searches for every object Git needs. The cost is linear in the number of packs and it applies to essentially every command.

git multi-pack-index write
ls .git/objects/pack/

multi-pack-index
pack-715bcb527ac3c833e86d1bdfcee36244987cc9ee.idx
pack-715bcb527ac3c833e86d1bdfcee36244987cc9ee.pack
pack-715bcb527ac3c833e86d1bdfcee36244987cc9ee.rev

One index covers all packs. A single lookup finds the object and tells Git which pack holds it.

git multi-pack-index verify

--preferred-pack=<pack> breaks ties when several packs contain the same object; without it, ties go to the pack with the lowest mtime.

core.multiPackIndex defaults to true, so a written index is used. As with the commit-graph, the question is whether anything writes it.

Expire and repack

The multi-pack-index enables incremental consolidation, which is what the incremental-repack maintenance task performs.

git multi-pack-index expire deletes pack files the index no longer references — packs whose objects have all been rewritten into newer packs.

git multi-pack-index repack selects several small packs and combines them into a larger one, updating the index. The selection targets a total at least the batch size; the default batch size of zero is a special case that attempts to repack everything into a single pack.

The two-step design avoids races. Expire removes only what the index says is unnecessary; repack adds a new pack and repoints the index before anything is deleted. Concurrent Git processes always have a consistent view.

This is the mechanism that replaces gc for the pack-sprawl problem, without gc’s all-at-once rewrite. See git maintenance at scale for why that trade is made.

Why many packs are slow

Worth being concrete about, because “many packs is bad” is repeated without explanation.

Each pack has an index sorted by object ID. Looking up an object in one pack is a binary search — fast, and logarithmic in the pack’s object count.

Git does not know which pack holds an object. So it searches them in turn. With one pack that is one binary search; with fifty it is up to fifty.

This applies to almost every command. Reading a commit, a tree, a blob — anything that touches the object database pays the multiplier. A command that reads ten thousand objects in a fifty-pack repository does five hundred thousand index lookups instead of ten thousand.

The multi-pack-index collapses that to one. A single sorted index over every object in every pack, with a pointer to the pack and offset. One binary search, then a direct read.

Which is why the pack count is a health metric. A repository whose pack count climbs steadily is one where nothing is consolidating, and its performance degrades in proportion.

Multi-pack bitmaps

git multi-pack-index write --bitmap

Reachability bitmaps precompute which objects are reachable from which commits. They are what makes serving a clone fast, because the server can answer “what objects does this client need” from a bitmap instead of a traversal.

A multi-pack bitmap covers the objects in the multi-pack-index rather than a single pack, which is what makes bitmaps usable on a repository that is not consolidated into one giant pack.

This matters most on the server side — a hosting service or a mirror serving many clones. On a developer’s machine the benefit is smaller, since they are not serving fetches.

--refs-snapshot=<path> supplies the reference tips taken before repacking, with a leading + marking preferred refs. This is machinery git repack uses internally; it is rarely invoked by hand.

Where these fit in the fetch path

Understanding when each structure is written explains why they are sometimes stale.

A fetch downloads a pack. That pack contains new commits, and the commit-graph does not know about them. Until something writes the graph, those commits are parsed the slow way — which is fine for a handful and slow for a large fetch after a holiday.

fetch.writeCommitGraph closes that gap by writing after every pack-downloading fetch. The manual notes that with --split, most executions create a very small file on top of the existing ones, occasionally merging them.

The maintenance prefetch task interacts with this well. Prefetch brings objects in hourly; the hourly commit-graph task indexes them. By the time the developer fetches, both the objects and the index are current.

Pushes do not affect the local graph in any way that matters — the commits were already local.

A rebase or a large local history rewrite creates commits the graph does not have. The graph is not wrong — it simply does not cover them, and Git falls back to reading objects for those. The next write picks them up.

None of this produces incorrect results. A stale commit-graph makes Git slower, never wrong. That property is what allows these to be caches written on a lazy schedule.

Diagnosing whether they are helping

The measurements that turn “we enabled it” into “it worked”.

Time git merge-base between two long-diverged branches, with and without the graph present. This is the cleanest single benchmark, it takes ten seconds to run, and merge-base underlies most higher-level operations.

Time git log --oneline -- <path> on a deep path, before and after changed-path Bloom filters. This is the developer-visible number on a monorepo.

Count pack files. ls .git/objects/pack/*.pack | wc -l. If it is one, the multi-pack-index is not doing anything for you; if it is fifty, it is doing a great deal.

Check the graph covers current history. A graph written a month ago on a repository with heavy traffic covers a fraction of what is now there. git commit-graph verify confirms integrity, not currency — the mtime of the file relative to recent commits is the better signal.

ls -l .git/objects/info/commit-graph* .git/objects/info/commit-graphs/ 2>/dev/null
ls .git/objects/pack/*.pack | wc -l
git log -1 --format=%cI

Compare the graph’s mtime to the newest commit date. A large gap means nothing has written it since, and the benefit has been eroding ever since.

The server side

Both structures matter more on a server than on a laptop, and are more likely to be neglected there.

A bare repository receiving pushes accumulates packs quickly and has no interactive user whose slowness would prompt investigation. It gets slower to serve, silently, for months.

Bitmaps are the server-specific benefit. Answering “which objects does this client need” is the expensive part of serving a clone, and a bitmap turns a graph traversal into a set operation. This is why multi-pack bitmaps exist — a large repository cannot be kept in one pack, and single-pack bitmaps therefore cannot cover it.

On GitHub this is handled for you and there is nothing to configure. On any mirror, cache or bare repository your organisation runs, it is yours:

git --git-dir=/srv/mirrors/monorepo.git commit-graph write --reachable --split
git --git-dir=/srv/mirrors/monorepo.git multi-pack-index write --bitmap

Schedule it. A mirror that is written once at setup and never again is the common failure. Either register it with git maintenance or run it from the host’s own scheduler.

Watch disk headroom. Writing a new pack and a new bitmap needs space alongside the existing ones. A host sized to fit the repository exactly cannot maintain it.

Operating these at scale

The practical guidance, given that both are caches nobody looks at.

Do not write them by hand on developer machines. Enable maintenance and let the commit-graph and incremental-repack tasks do it. Hand-run commands drift and stop.

Do write them explicitly on servers you operate. Mirrors, CI caches and bare repositories receive pushes and never run interactive commands, so nothing triggers a write.

Enable changed-path Bloom filters on large monorepos, and only there. On a small repository they cost more to build than they save.

Verify after anything unusual. After a filesystem problem, an interrupted maintenance run, or a repository copy, git commit-graph verify and git multi-pack-index verify are cheap and conclusive.

Deleting them is always safe. Both are derived. rm .git/objects/info/commit-graph* and rm .git/objects/pack/multi-pack-index lose nothing but speed, and both rebuild. This is the first remediation for any suspicion of corruption in either.

Note

Because these are caches, they are also a good first diagnostic. If a repository behaves strangely in history operations, delete the commit-graph and retry. If the behaviour changes, you have found a stale or corrupt graph; if it does not, you have eliminated it in thirty seconds.

A rollout plan for a large monorepo

Assume a repository with a million commits, a few hundred thousand files, and developers complaining that git log on a file is slow.

1. Establish the baseline. Time merge-base, log --oneline on a deep path, and status. Write the numbers down. Without them the rest is anecdote.

2. Write a plain commit-graph on one machine: git commit-graph write --reachable. Re-time. This alone usually resolves the merge-base and reachability complaints.

3. Add changed-path Bloom filters, bounded: git commit-graph write --reachable --changed-paths --max-new-filters=100000. Re-time the path-limited log. Repeat the write until the filters cover the history.

4. Confirm the split form is in use for ongoing writes, so incremental updates stay cheap.

5. Write a multi-pack-index and count packs before and after a subsequent incremental-repack.

6. Enable git maintenance start so all of this continues without anyone thinking about it, and stop running the commands by hand.

7. Do the same on every mirror and CI cache, from the host’s scheduler.

8. Re-measure a month later on a machine that has been in normal use. If the numbers have drifted back, maintenance is not running — check lastRun.

The order matters. Steps 2 and 3 are one-off wins you can demonstrate immediately; step 6 is what makes them permanent. Doing 6 first means nobody sees the improvement and nobody funds the rest.

What these do not fix

Both are frequently proposed as the answer to problems they have no bearing on.

They do not reduce repository size. The commit-graph and multi-pack-index are additional files. They make a large repository faster, not smaller.

They do not speed up cloning. A clone downloads objects; these are local indexes built afterwards. Partial clone is the clone-time lever.

They do not help with working-tree size. A checkout that writes 400,000 files is slow because of the filesystem, and no index changes that. Sparse checkout is the lever there.

They do not help with large files. A 300 MB binary is slow to transfer and slow to diff regardless. That is a storage strategy question.

They do not make git status fast on their own. Status is dominated by index size and filesystem scanning. The sparse index and filesystem monitor address that; the commit-graph does not.

Being clear about this matters because “we enabled the commit-graph and it did not help” is usually a case of it being applied to the wrong problem — the repository was slow for a reason these structures do not touch.

Common mistakes

Assuming they exist. Reading is on by default; writing is not automatic without maintenance.

Never enabling changed-path Bloom filters on a monorepo. The largest available win for path-limited history.

Building Bloom filters in one unbounded run on a huge repository. Use --max-new-filters to spread it.

Writing a non-split commit-graph on a schedule. Full rewrites every hour on a large history.

Leaving mirrors and CI caches without either structure. They degrade fastest and are checked least.

Treating a corrupt graph as repository corruption. Delete and rebuild; the repository is fine.

Running git commit-graph write with core.commitGraph disabled. It warns and does nothing, successfully.

Ignoring verify. Two cheap commands that answer a question people otherwise guess at.

Mental model

The commit-graph is an index of history’s shape; the multi-pack-index is an index across packs. Both are derived caches — deletable, rebuildable, and invisible when they work. On a large repository they are the difference between fast and unusable, and the only operational question is whether something is writing them on a schedule.

What you learned

• core.commitGraph and core.multiPackIndex default to true for reading; writing requires maintenance or explicit commands
• The commit-graph accelerates log, merge-base, --contains and every reachability question
• Generation numbers with corrected commit dates (version 2, the default) make traversal cutoffs correct despite bad timestamps
• Changed-path Bloom filters transform git log -- <path>, cost time to build, and are sticky once enabled
• Split commit-graphs write a small new layer instead of rewriting the whole file, which is what makes hourly maintenance viable
• The multi-pack-index replaces per-pack index lookups with one index across all packs
• expire and repack on the multi-pack-index are the incremental alternative to gc
• Multi-pack bitmaps mainly benefit servers, which serve fetches
• Both files are derived caches, safe to delete, and deleting is the first diagnostic

Exercise

Use a disposable local clone of a repository with real history. Everything here is local and reversible — the files created are derived caches you can delete at any point.

1. Check whether a commit-graph exists: ls .git/objects/info/. Predict: does it?

2. Time git merge-base between two long-diverged branches. Record it.

3. Run git commit-graph write --reachable. Time the same merge-base again. Compare.

4. Run git commit-graph verify. Predict: what does success look like?

5. Time git log --oneline -- <some/deep/path>. Then run git commit-graph write --reachable --changed-paths and time it again.

6. Delete the commit-graph files. Re-run the timings. Confirm the numbers return to the baseline.

7. Run git multi-pack-index write, then verify. Look at what appeared in .git/objects/pack/.

8. Count your pack files. If there is only one, explain why the multi-pack-index is not helping you yet.

Related lessons

Git Maintenance at ScaleWhat writes these on a schedule.

Repository Limits & HealthMeasuring the result.

Git ObjectsWhat these structures index.

Git Monorepos at ScaleWhere these matter most.