GPU Runners in GitHub Actions

Machine-learning code has the same need for CI as any other code, and rather more need for it — a model training script that crashes after four hours is an expensive way to discover a typo. The complication is that GPU capacity costs an order of magnitude more per minute than a standard runner, so the design question is not “how do we get a GPU” but “how little GPU time can we use”.

Before you start

• Self-hosted runners
• Matrix builds — separating cheap legs from expensive ones

Getting a GPU

There are three routes, and the right one depends on volume.

GitHub-hosted larger runners with GPU configurations are selected by a label you define when creating the runner in organisation settings:

runs-on: gpu-runner

Availability, GPU model and billing all depend on the account

GPU-equipped hosted runners are part of the larger-runners feature. Whether they are available to your organisation, which GPU models are offered, in which regions, and how they are billed all depend on the account’s plan and on organisation settings — and they change over time. Check Settings → Actions → Runners → Larger runners and your billing page for what your organisation actually has. Do not size a pipeline against a configuration you have not confirmed exists.

Self-hosted GPU runners — your own machines, or cloud instances you manage. Full control over the hardware, and full responsibility for drivers, CUDA versions and keeping the machine patched. See self-hosted runners, and note the warning there about never attaching self-hosted runners to public repositories.

External compute — the workflow submits a job to a training platform and polls for the result. The runner stays cheap; the accelerator is billed by whoever runs it. For long training jobs this is usually the right shape, because a six-hour job does not fit a runner’s timeout comfortably anyway.

Verify the GPU before using it

- name: Verify the GPU
  run: |
    nvidia-smi
    python -c "import torch; assert torch.cuda.is_available(), 'no CUDA device'; print(torch.cuda.get_device_name(0))"

What it doesConfirms a GPU is present and visible to the driver before the expensive part of the job starts.

Why we run itA job scheduled onto the wrong runner, or one whose driver failed to load, will otherwise fall back to CPU and run for hours producing a result nobody can trust — or crash forty minutes in.

Expected resultA table listing the device, driver and memory. A failure here is fast and unambiguous.

Fail fast and loudly. The silent CPU fallback is the expensive failure mode: everything appears to work, the job takes twenty times longer, and the numbers at the end are correct but the pipeline is useless.

Containers need explicit device access

A job running in a container does not see the host’s GPU unless the container runtime is told to pass it through:

jobs:
  train:
    runs-on: [self-hosted, gpu]
    container:
      image: nvcr.io/nvidia/pytorch:24.10-py3
      options: --gpus all
    steps:
      - uses: actions/checkout@v7
      - run: nvidia-smi

--gpus all requires the NVIDIA container toolkit installed on the host. Without it the container starts normally and nvidia-smi is simply not found — which reads like a missing package rather than a missing device.

The driver on the host and the CUDA runtime in the image must be compatible. Pinning the image tag matters more here than usual: a floating latest tag can pull an image whose CUDA runtime is newer than the host driver supports, and the failure message is not obviously about version skew.

Structuring the pipeline to use as little GPU as possible

This is where the real engineering is. The pattern that works splits the pipeline by cost:

1. Cheap gate, standard runner. Lint, type check, unit tests, and a tiny end-to-end run on synthetic data with a two-layer model. Catches the overwhelming majority of mistakes for pennies.

2. GPU smoke test, short timeout. One training step on a real device. Confirms the model builds, the data loader works and the loss is finite. Minutes, not hours.

3. Full GPU run, needs: the gate, and not on every commit. Scheduled nightly, triggered by label, or run on demand with workflow_dispatch.

jobs:
  fast-checks:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - run: make lint test-cpu

  gpu-smoke:
    needs: fast-checks
    runs-on: gpu-runner
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v7
      - run: nvidia-smi
      - run: python train.py --steps 10 --batch-size 2

  full-training:
    needs: gpu-smoke
    if: github.event_name == 'workflow_dispatch' || github.event_name == 'schedule'
    runs-on: gpu-runner
    timeout-minutes: 360
    steps:
      - uses: actions/checkout@v7
      - run: python train.py --config configs/full.yaml

needs: fast-checks is the load-bearing line. Without it, a syntax error costs GPU minutes.

timeout-minutes is a spend control on a GPU runner

The default job timeout is 360 minutes. On a GPU runner, a hung job burning that in full is a genuinely expensive accident — and a training script that deadlocks on a data loader will happily do it. Set an explicit timeout-minutes on every GPU job, sized to what the job should take rather than to the worst case you can imagine.

Caching model weights and datasets

Downloading a multi-gigabyte checkpoint on every run wastes GPU-priced minutes on network transfer.

- name: Cache model weights
  uses: actions/cache@v6
  with:
    path: ~/.cache/huggingface
    key: hf-${{ hashFiles('requirements.txt', 'configs/model.yaml') }}

Two limits shape this. The repository cache has a size quota, and a single large checkpoint can consume most of it and evict everything else. And datasets are usually far too large for the Actions cache — those belong in object storage the runner can reach, fetched with credentials obtained through OIDC.

On self-hosted runners a persistent local mount is simpler and faster than either, at the cost of the state-leakage properties discussed in self-hosted runners.

Determinism

GPU results are not bitwise reproducible by default. Non-deterministic kernel selection, atomic accumulation order and autotuning all contribute, which means a test asserting exact float equality will flake.

Either pin the sources of nondeterminism — seeding, deterministic algorithm flags, disabling autotuning — and accept the performance cost, or assert on tolerances instead of exact values. The second is usually the right call for CI; the first is worth it when you are debugging a genuine numerical regression.

Exercise

1. Structure an existing ML repository’s CI into the three tiers above. Measure what proportion of failures the cheap tier catches.

2. Add the nvidia-smi and torch.cuda.is_available() assertion as the first GPU step. Confirm it fails fast when scheduled onto a runner without a GPU.

3. Set timeout-minutes on every GPU job. Pick the number from observed durations, not from hope.

4. Add caching for model weights and measure the change in job duration.

5. Run the same training step twice and compare the loss values. If they differ, decide whether to pin determinism or assert on a tolerance.

Then what?

ARM runnersThe other specialised hardware case, and a cheaper one.

Self-hosted runnersOperating your own GPU fleet.

CachingQuotas, eviction and what does not belong in the cache.

EnvironmentsGating expensive jobs behind approval.