Automating a Repository Migration

A migration of four hundred repositories performed by hand is four hundred opportunities to miss a step.

The automation is not complicated. What makes it work is that it is driven by a manifest, resumable, verified automatically, and produces a report that anybody can read.

Before you start

• Migration planning — the project this executes
• API automation — the general discipline

The short answer

A manifest drives everything. One record per repository: source, destination, wave, owner, category, and the gap items to recreate.

A per-repository pipeline executes the steps in order, recording the outcome of each.

Idempotent and resumable. Re-running must be safe, and must not redo completed work.

Verified automatically. Commits, branches, LFS, permissions, and a successful CI run.

Reported continuously, because a fifteen-month project needs visible progress.

The manifest

The single source of truth for the migration, and the artifact everything else reads.

One record per repository, containing at minimum:

{
  "source": "old-platform/team-alpha/service-payments",
  "destination_org": "example-org-platform",
  "destination_name": "service-payments",
  "wave": 7,
  "owning_team": "payments",
  "category": "straightforward",
  "properties": { "tier": "production", "owning-team": "payments" },
  "gap": {
    "secrets": ["DEPLOY_TARGET", "REGISTRY_USER"],
    "environments": ["staging", "production"],
    "runners": false,
    "lfs": false
  },
  "status": "pending"
}

Version-controlled, so its changes are reviewed and its history explains decisions.

Generated from the inventory, not written by hand. See repository fleet management.

Status is updated by the tooling, and it is what makes the process resumable — a repository marked complete is skipped.

Excluded and out-of-scope repositories are in it too, with their category and reason. The manifest should account for every repository in the estate, so that “what happened to this one” always has an answer.

Building the manifest

The manifest is generated, and knowing what feeds it determines how good the migration’s planning is.

The source platform’s repository list, with sizes, activity dates and visibility.

Size measurements against the four importer limits — largest commit, largest file, repository size, metadata size. This is the preflight data, computed once during triage rather than repeatedly at wave time.

LFS usage, per repository, with an estimate of the content volume.

CI configuration, at least to the level of “has pipelines, how many, how complex”. This drives the conversion estimate, which is the largest cost item for a non-GitHub source.

Secrets and variables, by name. Values are never collected.

Environments and their protection rules.

Self-hosted runner usage.

Teams and permissions on the source, as input to the destination design rather than to be reproduced.

Ownership, derived from whatever source of truth exists — a service catalogue, CODEOWNERS, commit history as a last resort.

Assemble it once, review it with the teams, and version it. Teams will correct the ownership and the wave assignment, and their corrections are the most valuable input the manifest receives.

Regenerate the volatile fields periodically. Sizes and activity change over a fifteen-month project, and a repository that was straightforward in month one may exceed a limit by month ten.

The per-repository pipeline

The steps, each recording success or failure:

1. Preflight. Check size limits, confirm the destination organisation exists, confirm the destination name is free, confirm the source is reachable.

2. Migrate. Invoke the importer. Record the migration identifier and wait for completion.

3. Migrate LFS, if the manifest says the repository uses it.

4. Set custom properties, from the manifest. This is the moment the destination repository is classified, and doing it here rather than later means it is governed immediately.

5. Grant team access, from the manifest’s owning team.

6. Create environments with their protection rules.

7. Report the secrets to be provided, by name, to the owning team. Values are never handled by the tooling.

8. Verify. The automated checks below.

9. Update status in the manifest.

Each step is separately resumable. A repository that failed at step 6 should resume at step 6, not at step 1.

Log everything per repository, with timestamps and outcomes. When somebody asks about one repository three months later, the log is the answer.

Handling the gap items

The importer moves the repository; the automation handles what it does not. Each gap item has a different shape.

Secrets: report, do not transfer. The tooling lists the secret names the source repository has. The owning team provides values directly into the destination. This is a deliberate constraint, not a limitation — a tool that reads secret values will eventually log one.

Environments: create from a captured definition. Environment names, protection rules, required reviewers and branch restrictions are readable from the source and creatable on the destination. This one automates cleanly.

Variables: transfer directly. Unlike secrets, variable values are readable, and moving them is safe and worth automating.

Runners: re-register. Not something the tooling can do — the runner is a machine that must be pointed at the new destination. The automation’s job is to report which repositories expect self-hosted runners so nothing is missed. See runner groups.

Teams and permissions: create from the manifest. The destination team structure should come from your identity provider rather than from the source’s, which means this step reads the manifest rather than the source. See team synchronisation.

LFS: a separate migration step, with its own verification.

Webhooks: capture and recreate, minus their secrets, which follow the same rule as repository secrets.

Branch protection: do not recreate. The destination should have rulesets targeting by property, and a migrated repository with the right property is governed automatically. Recreating per-repository branch protection reproduces the source’s inconsistency. See enterprise rulesets.

That last point is worth emphasising. A migration is the chance to replace hundreds of hand-configured branch protections with a handful of targeted rulesets, and faithfully reproducing the old configuration throws that away.

Verification

Automated, per repository, and non-negotiable.

Commit count and tip SHA on the default branch, compared against the source.

Branch and tag counts, compared.

Repository size, compared, with a tolerance — packing differs between platforms.

LFS resolution. For repositories using LFS, clone and check out a commit referencing an LFS object. A pointer that does not resolve is the most common silent migration failure.

Issue and pull request counts, compared.

Permissions. The expected teams with the expected access.

Custom properties set as the manifest specifies.

A successful CI run is the real verification, and it cannot be fully automated because it requires the secrets. Mark it as a manual gate that the owning team completes.

Fail loudly. A verification failure should stop the wave for that repository and require a human decision, not be logged and skipped.

Idempotency and resumption

Re-running must be safe. A run interrupted at repository 240 of 400, restarted, must complete rather than duplicate.

Status in the manifest is the mechanism. Each step updates it; each run skips what is done.

Handle the partial states. A repository whose migration was initiated but whose completion was not recorded needs a check against the destination rather than a blind retry.

Never assume a step failed just because it errored. A timeout waiting for a migration does not mean the migration failed — check the destination before retrying, or you will attempt to migrate into a name that now exists.

Make the destination check the first thing preflight does on a retry.

The dry run

Every wave should be preceded by a dry run against the real manifest, and it catches most of what would otherwise fail during the window.

Preflight every repository in the wave. Size limits, destination availability, source reachability, team existence, property validity.

Report what would happen. Which repositories would migrate, which would be skipped and why, what the destination names would be.

Confirm the gap inventory. Which secrets, environments and runners each repository needs, so the team can prepare rather than discover during the window.

Check the destination is ready. Organisation exists, teams exist, rulesets in place, properties defined.

Run it a week before the wave, not on the day. The point is to find problems while there is time to fix them.

And run it again immediately before, because the estate changes — a repository grew, a name was taken, a team was renamed.

A dry run that reports no problems is not proof of success, but a dry run that reports problems has saved a window. The asymmetry justifies it every time.

Credentials

A dedicated migration identity, scoped to the operation.

Source credentials with read access to what is being migrated.

Destination credentials with the permission to create repositories and configure them.

Both in a secret manager, never in the manifest, never in a script, never in a log.

Exempt from conditional access where applicable — GitHub notes that service accounts should be exempt from Entra ID’s conditional access policy when using the importer, or the migration will be blocked.

Revoke both when the migration ends.

Danger

Never place real credentials in the manifest, in migration scripts, in logs, or in the repository holding the tooling. Use placeholders in examples and a secret manager in practice. A migration credential typically has broad read on the source and broad write on the destination — close to the largest grant available on either platform — and it is present for the duration of a long project.

And never handle secret values. The tooling reports which secrets a repository needs; the owning team provides the values directly into the destination. A migration script that reads secret values from one platform and writes them to another is a script that logs them somewhere eventually.

Reporting

A fifteen-month project needs visible progress, and the manifest already contains everything required.

Per wave: repositories planned, migrated, verified, failed, blocked.

Overall: the same, plus waves completed and remaining.

The tail: out-of-scope items with their reasons, and remediation items with their owners.

Failures, individually, with what failed and who is resolving it.

Generate it from the manifest on a schedule, and publish it where the whole organisation can see. A migration whose progress is invisible loses support long before it loses momentum.

And keep the historical reports. The record of when each repository moved is the answer to a surprising number of later questions.

Where the automation runs

A migration’s tooling runs for months and needs somewhere reliable to live.

A repository with the tooling and the manifest, version-controlled, reviewed. The manifest’s history is the migration’s decision log.

Waves triggered manually, not on a schedule. A migration wave has a freeze window agreed with a team, and it should start when a person says so.

Long-running steps need somewhere that tolerates them. A migration of a large repository can take a while, and a job runner with a short timeout will kill it. Either run it somewhere with a generous limit, or make the tooling initiate and poll rather than block.

Poll rather than block, generally. Initiate the migration, record the identifier, and check its status in a subsequent run. This makes the whole pipeline naturally resumable and removes the timeout problem.

State lives in the manifest, not in the job. A job that holds progress in memory loses it when it dies, and it will die.

Credentials from a secret manager, injected at runtime.

And keep the tooling simple. A migration is a temporary project, and elaborate tooling built for it is thrown away in eighteen months. Scripts that are readable and debuggable beat a framework, because the person debugging a failed wave at short notice may not be the person who wrote it.

Testing the tooling

Because the blast radius is the estate, and mistakes are visible.

A sandbox source and destination. Repositories you can migrate repeatedly, including ones with LFS, large files, many branches and unusual configurations.

Test the failure paths specifically. A migration that times out. A destination name that already exists. A repository exceeding a limit. A source that becomes unreachable mid-run. Each should produce a clear outcome rather than an ambiguous one.

Test resumption. Interrupt a run and restart it. Confirm it skips what is done and completes what is not.

Test idempotency. Run the same wave twice. Confirm nothing is duplicated and nothing errors unnecessarily.

Test the verifier against a deliberately broken migration. A verifier that passes everything has not been tested.

Run the pilot wave with the real tooling, not by hand. The pilot’s purpose is to test the process, and a pilot performed manually tests nothing about the automation that will run the other thirty-nine waves.

And keep the sandbox for the duration. Every change to the tooling mid-project should be tested there before it touches a real wave.

Common mistakes

No manifest. Nothing is resumable and nothing accounts for the whole estate.

A hand-maintained manifest. Wrong within a week; generate it.

Retrying without checking the destination. Duplicate or conflicting state.

No per-repository logging. “384 succeeded” without the other sixteen.

Verification by sampling. The failures are not evenly distributed.

Skipping LFS verification. The most common silent failure.

Handling secret values in the tooling. They will end up in a log.

Setting properties after the migration rather than during. A remediation campaign later.

No visible progress reporting. Support erodes on a long project.

Leaving the migration credential active. Standing access to two platforms.

Reference updating

The step that is easiest to automate and most often left manual.

Find the references. A fleet-wide code search for the source platform’s hostname, across the destination estate and anything not yet migrated. Workflow files, scripts, documentation, configuration.

Categorise them. A workflow checking out a migrated repository must be updated. A documentation link can rely on redirects for a while. A hardcoded clone URL in a deployment script must be updated before the source is decommissioned.

Automate the mechanical updates. A bulk pull request replacing old URLs with new ones, per repository, is exactly the kind of change that automates well — small, uniform, and reviewable at a glance. See API automation.

Do it per wave, so the work is distributed and each team reviews their own changes while the migration is fresh.

Track the remaining count. A query for the old hostname across the estate, run weekly, with the number trending to zero. That number is the readiness metric for decommissioning.

And check outside the estate. Deployment systems, monitoring, ticketing, external services. These are not in a code search and they need enumerating during triage.

Redirects are a grace period, not a solution. They work until the source is switched off, which is exactly when the remaining references break — and by then the migration team has moved on.

Reporting the tail

The part of the reporting that matters most in the second half of the project.

Every out-of-scope item, individually, with its reason and its owner. Not a count — a list, because each one is a decision somebody made and may need to defend.

Every remediation item, with its blocker and its expected date.

Every team that has not started, with the reason and the escalation status.

Age. How long each tail item has been open. An item open for four months with no movement is not a work item; it is an unresolved decision.

And the readiness metric: what remains before the source can be made read-only. That single number, published weekly, is what converts a diffuse tail into something with an end.

Report the tail from week one, not from the point at which it becomes a problem. A tail item identified in triage and tracked from the start is manageable; the same item discovered in month twelve is a crisis.

The manifest already has all of this. Every field needed for tail reporting is a field the manifest carries, which is the argument for making it comprehensive at the start rather than minimal.

What to do when a repository fails

A failure during a wave needs a decision quickly, and having the options enumerated in advance shortens it.

Retry, after checking the destination. Transient failures happen. The check first is what prevents a duplicate.

Defer to the next wave. A repository with a problem that will take days to resolve should not hold up the rest of its team’s migration. Move it, mark it, and continue.

Move it to the remediation category. Exceeds a limit, has an LFS problem, or has something unusual. It becomes a tail item with an owner.

Declare it out of scope. If the repository is dead, or the effort is disproportionate, archive it on the source and record the decision.

Roll back the whole wave. Rarely necessary for a single repository failure, and appropriate when a systemic problem is affecting many.

The decision should take minutes, not a meeting. The wave’s owner should have the authority to choose any of these, and the manifest records which was chosen.

Record the reason in the manifest, always. A repository deferred with no recorded reason becomes a mystery in month ten, and mysteries do not get resolved — they get deferred again.

And feed failures back into the process. A failure mode seen twice should be caught by preflight the third time.

After the last wave

The automation has a job after the migrations finish, and it is worth building rather than doing by hand.

Reconcile source against destination. Every repository in the manifest, confirmed present at its destination or accounted for as out of scope. A final full pass, not a sample.

Confirm the properties are set on every migrated repository, because this is what determines whether targeted governance applies. Any repository that arrived without them is a gap.

Confirm permissions. Every repository has its owning team with the expected access, and no repository has unexpected grants inherited from the migration identity.

Find repositories on the destination that are not in the manifest. Created during the migration, outside the process. Each needs classifying.

Report remaining references to the source platform, as the readiness signal for making it read-only.

Then revoke the migration credentials, on both platforms, and confirm they no longer work.

Archive the tooling with its manifest. In eighteen months somebody will ask when a repository moved, or why one was excluded, and the manifest is the only record.

And run one final report — the complete account of the migration, per repository, with what happened to each. That document is what closes the project, and it is generated rather than written.

Mental model

Migration automation is a manifest, a pipeline and a verifier. The manifest accounts for every repository and records what happened to each; the pipeline executes the steps idempotently; the verifier proves the result. Everything hard about a migration at scale is a consequence of not having one of those three.

What you learned

• A generated, version-controlled manifest accounting for every repository is the foundation
• The per-repository pipeline should be separately resumable at each step
• Set custom properties during migration so repositories arrive governed
• Verification must cover commits, branches, LFS resolution, permissions and properties, with a CI run as a manual gate
• LFS pointers that do not resolve are the most common silent migration failure
• Never retry blindly after a timeout — check the destination first
• The tooling should report which secrets are needed, never handle their values
• Migration credentials are unusually powerful and must be scoped, protected and revoked
• Progress reporting generated from the manifest is what sustains a long project

Exercise

Reason from an estate you know, or build against a sandbox.

1. Design the manifest schema. What fields does each record need?

2. Generate a manifest for one organisation from its repository inventory.

3. List the pipeline steps for a repository in your estate. Which are separately resumable?

4. Design the verification checks. Which can be automated, and which need a human?

5. Write the preflight check for size limits against all four importer limits.

6. Describe what your tooling does when a migration times out. Predict: would a blind retry be safe?

7. Determine how secret values would reach the destination without the tooling seeing them.

8. Design the weekly progress report generated from the manifest.

Related lessons

Migration PlanningThe project this executes.

GitHub Enterprise ImporterThe tool and its gap.

API AutomationThe general discipline.

Repository Fleet ManagementWhere the manifest comes from.