Large Binary Files and Git: A Storage Decision Guide

Most large files in Git repositories are there because somebody committed one once and nobody removed it.

That is not a criticism — it is the default outcome of a system where adding a file is one command and removing it from history is a coordinated organisational operation. This article is the decision that should have happened first.

Before you start

• Git LFS — the mechanism, if the answer is LFS
• Repository limits and health — what happens when you get this wrong

The question that decides it

Not “how big is it”. What is it?

Does checking out an old commit need to give you the matching version of this file?

If yes, the file is an input — it changes alongside the code and the pairing matters. Git or Git LFS.

If no, the file is an output or a reference, and it belongs in a system built for artifacts.

That single question resolves most cases, and it is a better guide than size. A 200 MB test fixture that must match the code reading it belongs in LFS. A 2 MB compiled binary that a build produces belongs nowhere near version control.

Why generated binaries must not accumulate

The failure mode, stated once, because it is the most common and the most expensive.

Every version is retained forever. Git does not delete. A binary committed weekly for three years is 156 copies in history, all downloaded by every clone, permanently.

Binaries do not delta. Packfiles compress similar objects against each other. Recompiling changes the output throughout, so each version costs full size.

Deleting does not help. Removing the file removes it from the working tree; the objects remain reachable from history and remain in every clone.

The growth is invisible until it is severe. Nobody notices a repository growing 400 MB a quarter until a clone takes twenty minutes and somebody asks why.

And the file was reproducible the whole time. A build output can be regenerated from source. Storing every historical version of something you can recreate is paying permanent storage and bandwidth for zero information.

Danger

The specific pattern to look for: a dist/, build/, bin/ or target/ directory that is tracked. It is almost always a mistake, it is almost always old, and it is almost always the largest single contributor to repository size. git count-objects -vH plus a look at the largest blobs finds it in two minutes.

The options

Seven places a large file can live, with what each is genuinely for.

Git. Versioned with the source, full history, every clone gets everything. Correct for source code and small assets. GitHub enforces a 100 MB limit on individual objects, so anything larger cannot go here regardless.

Git LFS. Versioned with the source, content stored separately, fetched on demand. Correct for large inputs that must match the code. Costs storage and bandwidth billing, and a client every developer needs.

A package registry — npm, PyPI, Maven, NuGet, or GitHub Packages. Correct for versioned libraries other code depends on. Provides semantic versioning, dependency resolution and immutability.

Release assets. Files attached to a Git tag. Correct for distributable outputs of a release — installers, compiled binaries, signed archives. Immutable in practice, downloadable without a clone.

Object storage — S3, Azure Blob, GCS. Correct for large datasets, media libraries and anything referenced by URL and checksum. Cheap, unlimited, and provides no versioning semantics unless you build them.

An artifact repository — Artifactory, Nexus, or similar. Correct for organisations that already run one, with retention policies, promotion and access control across artifact types.

A container registry. Correct for container images and, increasingly, other OCI artifacts. Immutable by digest, with the tooling already in your pipeline.

The decision table

File type	Belongs in	Why
Source code	Git	It is the thing being versioned
Small images, icons, fonts	Git	Small, change with the code, need pairing
Design source files (PSD, Sketch)	Git LFS	Large inputs that must match a release
Test fixtures that must match code	Git LFS	Pairing matters, and they are inputs
Large media used by the build	Git LFS or object storage	Depends whether an old commit must resolve
Compiled binaries	Release assets	Outputs of a specific release
Installers	Release assets	Distributable, immutable, downloadable
Libraries other code imports	Package registry	Dependency resolution is the point
Container images	Container registry	Purpose-built, digest-addressed
Machine-learning models	Object storage or a model registry	Large, versioned separately, referenced
Large datasets	Object storage	Too large for Git, referenced by checksum
Database exports	Object storage	Frequently sensitive; retention matters
Archives and backups	Object storage	Never version control
Documentation builds	Nowhere	Regenerate them
Vendored dependencies	A package manager	Unless you have a specific reason

Working through the categories

The cases people actually ask about, with the reasoning rather than just the answer.

Machine-learning models

Not Git, and usually not LFS. Model weights are large, they change often, and their relationship to source is looser than it appears — the same code trains many models, and the same model serves many code versions.

Object storage with a versioning scheme, or a purpose-built model registry. The repository holds the training code and a reference: a URI plus a checksum.

Why not LFS? Model files reach sizes where the per-file limit becomes a real constraint, the bandwidth cost of every CI run fetching them is substantial, and LFS gives you nothing a versioned object store does not — while adding a client dependency and a quota.

The reference in the repository is the important part. models/production.json containing a URI and a SHA-256 is version-controlled, diffable, reviewable, and tiny.

Media and design assets

The one case where LFS is frequently correct. A design file that a build consumes, where checking out last quarter’s release should give you last quarter’s assets.

Object storage instead if the media is a library rather than a build input — a marketing asset collection referenced by URL is not something an old commit needs to resolve.

Consider file locking for unmergeable formats, covered in the LFS article.

Installers and compiled binaries

Release assets, essentially always. They are the output of a tagged release, they need to be downloadable by people who will never clone the repository, and they should be immutable.

Never in Git history. This is the category that causes the most damage, because installers are large, they are produced on every release, and the repository grows monotonically.

Archives and database exports

Object storage, and think about whether they should exist at all. A database export is frequently sensitive, and putting one in a repository — where it is readable by everybody with access and permanent in history — is a disclosure with no undo.

If it is test data, generate it or use a small anonymised fixture.

Danger

A database export committed to a repository is a data exposure that history rewriting does not fully undo — the objects remain in every existing clone and every fork. If you find one, treat it as an incident: assess what it contained, follow your data breach process, and only then consider history cleanup. The credential rotation ordering logic applies — the exposure is already real.

Vendored dependencies

Usually a package manager. Committing node_modules/, vendor/ or equivalent puts thousands of files and megabytes into history for something a lockfile already pins deterministically.

The legitimate exceptions are real: an air-gapped build, a dependency whose registry cannot be relied upon, or a supply-chain requirement to have the exact bytes reviewed. Those are decisions with reasons, and they should be written down so the next person does not undo them.

Documentation and generated output

Regenerate it. Built HTML, generated API references, compiled diagrams. If it can be produced by a command, the command belongs in the repository and the output does not.

The exception is a published site served from a branch, which is a deliberate deployment mechanism rather than accidental accumulation.

Large test fixtures

The genuinely ambiguous case. A 400 MB corpus that a parser test reads is an input, it must match the code, and by the deciding question it belongs in LFS.

But ask whether it must be that large. Most large fixtures are large because somebody captured real data once, not because the test needs the volume. A 2 MB representative sample usually exercises the same paths.

And ask whether every developer needs it. If only a nightly job runs the large-corpus tests, the corpus can live in object storage and be fetched by that job. Developers get a fast clone; the coverage still happens.

The pattern that works: a small fixture in the repository for the tests everyone runs, a large corpus outside it for the tests CI runs, and a documented command to fetch the corpus locally when someone needs to reproduce a failure.

Finding what is already there

Before designing a policy, find out what you have.

# Overall size and object count
git count-objects -vH

# The largest objects in history, with their paths
git rev-list --objects --all \
  | git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' \
  | awk '$1=="blob" {print $3, $4}' \
  | sort -rn | head -20

That second command is the one to run first on any repository somebody has called slow. It lists the largest blobs with their paths, and the answer is usually immediately obvious — a dist/ directory, a committed installer, a dataset somebody added in 2021.

git-sizer reports every dimension at once and is worth installing for this purpose.

Expect surprises. The largest contributor is frequently something nobody remembers adding, in a directory nobody looks at, from a commit whose author has left.

Deciding what to do about it

Finding the problem is easy; remediating it is a choice with costs.

If the file is still needed and belongs elsewhere: move it, add a reference, add a .gitignore rule so it cannot recur. The repository does not shrink — history still contains it — but growth stops.

If growth has stopped and the size is tolerable: that may be the whole remediation. A 3 GB repository that is no longer growing is workable with partial clone, and history rewriting is a large organisational cost.

If the size is genuinely unworkable: history rewriting is the only thing that shrinks it, and it invalidates every clone, every open pull request and every recorded commit SHA. The LFS article has the coordination sequence; repository health covers when it is warranted.

The ordering that matters: stop the growth first, then decide about history. Stopping growth is a .gitignore change and a build fix. Rewriting is a project, and doing it while files are still being committed means doing it twice.

The cost comparison nobody makes

The reason bad choices persist is that Git storage feels free at the point of decision and the cost lands on everybody else later.

Git and LFS storage is charged to the organisation; clone time is charged to every developer, every CI job, every day. A 4 GB repository that could be 400 MB costs a few dollars of storage and thousands of engineer-minutes a year. The storage line item is the small number.

Object storage is cheaper per gigabyte than either, and it expires things. That is the whole comparison for anything that is not versioned with source.

Release assets and package registries have their own billing, which varies by plan and by whether the repository is public. Check current pricing rather than assuming — it changes, and the assumption that one option is free is frequently the reason a wrong choice was made.

The number worth computing before a large migration: how many clones per week, times the size difference, times the wall-clock cost. That figure is what justifies the remediation project, and it is usually larger than people expect.

Note

Storage and bandwidth pricing for Git LFS, GitHub Packages and Actions changes over time and differs by plan. Look up current figures in your organisation’s billing settings rather than working from remembered numbers.

Preventing recurrence

Policy beats remediation.

.gitignore for build output, checked into the template so new repositories inherit it.

A pre-commit hook or a CI check that fails on files over a threshold. A repository that rejects a 50 MB commit at push time never acquires the problem.

Push rulesets can restrict file paths and, on GitHub, block pushes containing files above a size. That is enforcement rather than convention.

Make the alternative easy. People commit build outputs because there is no obvious place to put them. A documented artifact store and a working publish step removes the motivation.

Review the largest blobs periodically. A quarterly run of the command above catches accumulation while it is still cheap to fix.

Referencing external artifacts well

Once a file lives outside the repository, the reference becomes the thing under version control — and a bad reference is worse than no policy.

Reference by identity, not by location alone. A URL says where; a checksum says what. A reference containing both is verifiable:

{
  "model": "fraud-detection",
  "version": "2026-08-14",
  "uri": "s3://example-models/fraud-detection/2026-08-14/weights.bin",
  "sha256": "0000000000000000000000000000000000000000000000000000000000000000",
  "size_bytes": 4831838208
}

Verify the checksum on fetch. A download step that does not check what it got is a download step that will one day get something else. This is the same reasoning as pinning a container image by digest.

Make the reference immutable. A URI pointing at latest/weights.bin reintroduces exactly the mutability that moving the file out of Git was supposed to control. Version the path, or address by content.

Keep the reference small and diffable. A JSON file with a handful of fields shows in a pull request as a readable change. A binary manifest does not.

Record how to fetch it. A script or a documented command, in the repository. A reference nobody can resolve is a reference that will be resolved by somebody guessing.

And handle the retention question. Object storage does not expire things unless you tell it to, and an unversioned bucket that accumulates every model ever trained has replaced a Git problem with a billing one. Retention policy is part of adopting external storage, not an afterthought.

Common mistakes

Committing build outputs. The most common and most expensive.

Reaching for LFS as the default answer for anything large. It is one option and it is a permanent operational commitment.

Assuming deleting a file shrinks the repository. It does not; history retains it.

Committing datasets or database exports. Frequently a data exposure as well as a size problem.

Vendoring dependencies without a reason. Thousands of files a lockfile already pins.

Rewriting history before stopping the growth. You will do it twice.

No size limit at push time. The problem recurs.

Storing models in Git or LFS. Wrong tool; the reference belongs in the repository and the weights do not.

Treating this as a storage decision only. A database export in a repository is a security incident, not a size issue.

Mental model

Version control is for inputs that change with the code. Artifact stores are for outputs and for large references. The question is not how big the file is — it is whether an old commit needs to resolve to the matching version.

Everything follows. Design files pair with releases, so they are versioned. Compiled binaries are produced by releases, so they are attached to them. Datasets are referenced by identity, so a checksum in the repository is enough.

What you learned

• The deciding question is whether an old commit must resolve to the matching file, not the file’s size
• Git retains every version forever and does not delta binaries, so binary history compounds permanently
• Build outputs, installers and archives belong in release assets or object storage, never in Git
• Machine-learning models belong in object storage or a model registry, with a reference in the repository
• LFS is correct for large inputs that must pair with the source, and is a permanent operational commitment
• Deleting a file does not shrink the repository; only history rewriting does
• Find the problem with a largest-blobs listing before designing any remediation
• Stop the growth before rewriting history, or you will rewrite twice
• A database export in a repository is a data exposure, not a storage problem

Exercise

Use a disposable repository. Synthetic files only — no real data.

1. Create a repository. Add a dist/ directory with a 10 MB generated file. Commit it, regenerate it and commit again, three times.

2. Run the largest-blobs command above. Predict: what does it show?

3. Delete dist/ and commit the deletion. Run git count-objects -vH. Predict: did the repository shrink?

4. Add dist/ to .gitignore. Try to commit a new file there. Predict: does it stop you if the file is already tracked?

5. Run git rm -r --cached dist/ first, then repeat. Compare.

6. Take a real repository you work on and run the largest-blobs command. Classify the top five: input, output, or reference.

7. For each output you find, write down where it should live instead.

8. Delete the disposable repository.

Related lessons

Git LFSThe mechanism, when LFS is the answer.

Repository Limits & HealthThe limits large files push you toward.

Container Image TaggingArtifact identity done properly.

Removing Secrets from Git HistoryIf what you found was sensitive.