Notes about buildkit

I have recently discovered tool called Rancher Desktop. It’s alternative to Docker Desktop solution. One thing that interest me much in Rancher Desktop was how container images are built, it’s handled by tool kim - The Kubernetes Image Manager. KIM is built around buildkit, which was released quite some time ago, but I haven’t got chance to closer look at it. Buildkit is separate tool used to build container images in much efficient way it could be intergrated with docker, it’s quite similar to kaniko. This post it’s not complete overview with lot’s of hints and caveats, but rather subjective list of things to consider and test:

• buildkitd can be exposed in different ways ie. unix socket, tcp socket, you can start it as k8s pod and even loadbalance your container image build requests between using consistent hashing algorithm
• support multiple ways of exposing image build artifacts ie. remote image registry, docker itself, tarball oci image
• caching layers with garbage collection using remote image registry (should support cache layers manifest) or local dirs (faster)
• caching using --mount=type=cache in Dockerfile, more info
• securely dealing with sensitive data by using --mount=type=secret in Dockerfile, more info
• multi-platform builds
• can be started in rootless mode, but using this mode enforce you to disable apparmor and seccomp, root mode enforce using privileged mode

Time savings with using--mount=type=cache:

$ docker buildx create --use # run container with buildkitd
$ cat Dockerfile
# syntax=docker/dockerfile:1

FROM ubuntu:focal
RUN apt update && apt-get --no-install-recommends install -y wget ca-certificates
RUN --mount=type=cache,target=/tmp \
  [ -f /tmp/linux-5.14.13.tar.xz ] || wget -O /tmp/linux-5.14.13.tar.xz  https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.14.13.tar.xz

$ docker buildx build -f Dockerfile -t buildkit:mountcache --load --progress plain  .
...
DONE 24.8s

$ docker buildx build -f Dockerfile -t buildkit:mountcache --load --progress plain  . # second run with slightly changed layer with wget to enforce rebuild it
...
DONE 2.3s

$ docker diff buildx_buildkit_compassionate_cori0 # ensure that linux-5.14.13.tar.xz was placed in cache
C /root
A /root/.ash_history
C /var
C /var/lib
A /var/lib/buildkit
C /run
A /run/buildkit
A /run/buildkit/buildkitd.sock
A /run/runc

$ docker exec -ti buildx_buildkit_compassionate_cori0 sh -c "find /var/lib/buildkit -type f -iname linux-5.14.13.tar.xz"
/var/lib/buildkit/runc-overlayfs/snapshots/snapshots/96/fs/linux-5.14.13.tar.xz