Configuring Docker log file size

I was looking at drive utilization, and thought that my /mnt/docker/ directory was a bit bigger than I thought it should have been, so I started doing some some digging.

I found that I had a rather large log file from my Gitlab Docker container.

ls -lh *-json.log
-rw-r----- 1 root root 13G Jan 22 13:42 638615a4624620ea0ec3fb0fc0c1f25a51c3c05aa68a92cc0b943dcf0476cf86-json.log

Turns out that the Gitlab Docker container doesn't rotate the log out of the box.

Fixes for large Docker log file

There are a couple of different ways to address a large Docker log file.

One way is a quick, temporary fix, the other two require configuration changes.

Use truncate to truncate the file

Quick, but temporary fix, is to simple truncate the log file:

truncate -s 0 /mnt/docker/containers/638615a4624620ea0ec3fb0fc0c1f25a51c3c05aa68a92cc0b943dcf0476cf86/*-json.log

Fixes the issue, but doesn't stop it from coming back again.

Change the default Docker container configuration

This option impacts all Docker containers, now, and in the future.

Edit or create the file /etc/docker/daemon.json file and add as options:

"log-opts": {
  "max-size": "10m",
  "max-file": "10"
}

Since I moved the Docker root directory recently, and specified using the ZFS driver, my /etc/docker/damon.json file would look like this:

{
  "data-root":"/mnt/docker",
  "storage-driver": "zfs",
  "log-opts": {
    "max-size": "10m",
    "max-file": "10"
}

Restart the Docker deamon to use the new settings.

Change the Gitlab Docker container configuration

This is the solution I used

In the docker-compose yaml file, we can simply specify the maximum size and number of log files allowed.

Under the service configuration add:

  logging:
    opts:
      - max-size: 10m
      - max-file: 3

Redeploy the Gitlab container to use the new settings.

Results

A couple of days after implementing the change in the docker-compose file:

ls -lh *-json.log*
-rw-r----- 1 root root 4.2M Jan 24 08:36 638615a4624620ea0ec3fb0fc0c1f25a51c3c05aa68a92cc0b943dcf0476cf86-json.log
-rw-r----- 1 root root 9.6M Jan 24 07:39 638615a4624620ea0ec3fb0fc0c1f25a51c3c05aa68a92cc0b943dcf0476cf86-json.log.1
-rw-r----- 1 root root 9.6M Jan 24 05:25 638615a4624620ea0ec3fb0fc0c1f25a51c3c05aa68a92cc0b943dcf0476cf86-json.log.2

References

Gitlab Docs - GitLab Docker images https://docs.gitlab.com/ee/install/docker.html#docker-containers-exhausts-space-due-to-the-json-file

Docker Docs - Manuals / Docker Engine / Logging / Container logs / Configure logging drivers https://docs.docker.com/config/containers/logging/configure/