Automatic Cleanup of Docker Cache Volumes for GitLab Runner

The problem

When using the Docker executor, GitLab Runner creates volumes to store the cache between builds. However, these volumes have no built-in mechanism for automatic cleanup, and over time they can grow to enormous sizes – it is not unusual to see hundreds of gigabytes or even terabytes of occupied space.

You can list the volumes with:

docker volume ls | grep runner

And find out their size with:

for vol in $(docker volume ls -q | grep -E '^runner-.*-cache'); do
    SIZE=$(docker run --rm -v "$vol":/data alpine du -sh /data 2>/dev/null | cut -f1)
    echo "$SIZE    $vol"
done

Typical output may look like this:

132G    runner-yd6eca4h-project-100-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70
124G    runner-88bmy4dg2-project-100-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70
29G     runner-0558e6a41bfb19b22a086fc2330d186c-cache-3c3f060a0374fc8bc39395164f415a70

Notice the two different naming formats – older versions of GitLab Runner (before 15.x) used a 32-character hash, while newer versions use the format runner-{system_id}-project-{id}-concurrent-{n}.

The cause

GitLab Runner creates new cache volumes when:

• the cache:key in .gitlab-ci.yml changes
• the runner is re-registered (new system_id)
• the project or branch changes (depending on the cache key configuration)

The old volumes, however, stay on the disk and are never deleted automatically. On top of that, re-registering the runner generates a new system_id, and all previous volumes become "orphaned".

You can find the current system_id of your runner with:

cat /etc/gitlab-runner/.runner_system_id

The solution

One-time cleanup

GitLab Runner ships with an official script for clearing the cache. On Debian/Ubuntu it is part of the gitlab-runner package and you will find it at /usr/share/gitlab-runner/clear-docker-cache.

/usr/share/gitlab-runner/clear-docker-cache

The script offers several options:

# Show help
/usr/share/gitlab-runner/clear-docker-cache help

# Removes only dangling volumes (default)
/usr/share/gitlab-runner/clear-docker-cache prune-volumes

# Removes all runner cache volumes
/usr/share/gitlab-runner/clear-docker-cache space

The official script contains a workaround for a behavior change in Docker 23.0 that artificially lowers the API version to 1.41. With newer Docker versions (API 1.44+), this causes an error, for example:

Error response from daemon: client version 1.41 is too old. Minimum supported API version is 1.44

In that case, use Docker commands directly:

# Removes volumes with the GitLab Runner label
docker volume prune -f --filter "label=com.gitlab.gitlab-runner.managed=true"

To manually delete all cache volumes:

docker volume ls -q | grep -E '^runner-.*-cache' | xargs -r docker volume rm

Automatic cleanup with cron

For regular cleanup, create the file /etc/cron.d/gitlab-runner-cache-cleanup:

# Every Sunday at 3:00, removes unused runner cache volumes
0 3 * * 0 root docker volume prune -f --filter "label=com.gitlab.gitlab-runner.managed=true" >> /var/log/gitlab-runner-cache-cleanup.log 2>&1

Prevention

To limit cache growth directly in your projects, you can set an expiration in .gitlab-ci.yml:

cache:
  key:
    files:
      - package-lock.json  # new cache only when dependencies change
  paths:
    - node_modules/
  expire_in: 1 week

However, the expire_in setting only works at the GitLab server level, not at the level of Docker volumes. To actually limit the size at the runner level, you need to use the cleanup described above.

Further resources

• GitLab Runner Docker executor documentation
• Clearing the cache
• GitLab CI/CD cache configuration