How overlay filesystems work, and what to watch out for
/ 7 min read
Table of Contents
I looked into overlay filesystems while working on fast container startup for
short workloads. A container image is a stack of layers, and overlayfs is
what turns them into a single root filesystem for the container, so I started
there before looking at the alternatives, like lazy loading of images.
An overlay mount combines one or more read-only lower directories with a writable upper directory, and shows the result in a merged directory. Reads come from whichever layer has the file, and writes always go to the upper directory, so the lower ones are never modified.
All the examples below run as root, in an empty directory (/tmp/ovl here).
Mounts themselves are covered in the
post on mount namespaces.
A first overlay mount
An overlay needs four directories: the lower layer, the upper layer, a work directory that overlayfs uses internally, and the mount point:
mkdir lower upper work mergedecho "from the lower layer" > lower/hello.txt && echo bye > lower/old.txtmkdir lower/conf.d && touch lower/conf.d/a.conf lower/conf.d/b.confmount -t overlay overlay -o lowerdir=lower,upperdir=upper,workdir=work mergedls merged; ls upperconf.dhello.txtold.txtThe merged view shows the lower layer, and the upper layer is still empty. The lower directories must not be modified while they are part of a mounted overlay, which is why all the files are created first.
Copy-up
Let’s modify a file from the lower layer through the merged view:
echo "changed in merged" > merged/hello.txtcat lower/hello.txt; cat upper/hello.txtfrom the lower layerchanged in mergedThe first write to a lower file copies it up: the whole file is copied to the upper directory, then modified there. The lower file is untouched.
A detail that is easy to get wrong when working out where a file came from: the inode number in the merged view doesn’t change after the copy-up.
stat -c "%i %n" lower/hello.txt upper/hello.txt merged/hello.txt680228 lower/hello.txt680234 upper/hello.txt680228 merged/hello.txtWhen the layers are on the same filesystem, overlayfs keeps showing the lower file’s inode number, so that it stays stable for programs that rely on it. To know which layer a file comes from, look in the upper directory instead.
Deleting files: whiteouts
A file from the lower layer can’t be deleted, since the lower layer is never modified. Instead, overlayfs creates a whiteout in the upper directory:
rm merged/old.txtls merged; ls -l upper/old.txtconf.dhello.txtc--------- 2 root root 0, 0 Oct 31 11:07 upper/old.txtA whiteout is a character device with major and minor numbers 0, 0. When
overlayfs sees one in the upper layer, it hides the file with the same name in
the lower layers.
Replacing directories: opaque directories
Deleting a directory and creating a new one with the same name is a bit different:
rm -r merged/conf.d && mkdir merged/conf.d && touch merged/conf.d/new.confls merged/conf.d; ls lower/conf.dnew.confa.confb.confOnly new.conf is visible, even though a.conf and b.conf are still in the
lower layer. The new directory in the upper layer is marked as opaque, with an
extended attribute, which tells overlayfs not to look into the lower layers
for its content:
python3 -c 'import os; print(os.getxattr("upper/conf.d", "trusted.overlay.opaque"))'b'y'(getfattr -n trusted.overlay.opaque upper/conf.d shows the same thing, if
the attr package is installed.)
umount mergedSeveral lower layers
lowerdir accepts a list of directories separated by :, and the leftmost one
is on top:
mkdir layer1 layer2 upper2 work2 merged2echo "from layer1" > layer1/app.conf && echo "from layer2" > layer2/app.conf && echo "only in layer2" > layer2/extra.confmount -t overlay overlay -o lowerdir=layer1:layer2,upperdir=upper2,workdir=work2 merged2cat merged2/app.conf merged2/extra.conffrom layer1only in layer2umount merged2This is how container images are mounted: each image layer is a lower
directory, and the container gets its own upper directory. Docker’s overlay2
storage driver keeps the layers under /var/lib/docker/overlay2/ and supports
up to 128 lower layers.
Image layers are tar files, which can’t contain overlayfs whiteouts. The OCI
image format represents a deleted file as an empty file named with a .wh.
prefix (.wh.old.txt), and an opaque directory with a .wh..wh..opq file.
These are turned into whiteouts and opaque directories when the layers are
unpacked for overlayfs.
Which filesystems can be used
The work directory must be on the same filesystem as the upper directory. The lower layers can be on other filesystems, for example a tmpfs:
mkdir mem && mount -t tmpfs mem mem && mkdir mem/lower mem/work && echo "on tmpfs" > mem/lower/xstat -f -c "%T %n" mem/lower upper2mount -t overlay overlay -o lowerdir=mem/lower,upperdir=upper2,workdir=work2 merged2 && cat merged2/xtmpfs mem/lowerext2/ext3 upper2on tmpfsBut with the work directory on the tmpfs and the upper directory on ext4, the mount fails, with a message that doesn’t say why:
umount merged2mount -t overlay overlay -o lowerdir=lower,upperdir=upper2,workdir=mem/work merged2mount: /tmp/ovl/merged2: wrong fs type, bad option, bad superblock on overlay, missing codepage or helper program, or other error. dmesg(1) may have more information after failed mount system call.The actual reason is in the kernel log (dmesg). The upper filesystem also
has to support extended attributes, which rules out some network filesystems.
Copy-up gotchas
Copy-up copies the whole file, even when only the metadata changes. A chmod
on a 100 MB file:
mkdir l3 u3 w3 m3 && head -c 100M /dev/zero > l3/big.binmount -t overlay overlay -o lowerdir=l3,upperdir=u3,workdir=w3 m3chmod +x m3/big.bin && du -sh u3101M u3With the metacopy=on option, only the metadata is copied up, and the data is
still read from the lower layer until the file is written to:
umount m3 && rm -rf u3 w3 && mkdir u3 w3mount -t overlay overlay -o lowerdir=l3,upperdir=u3,workdir=w3,metacopy=on m3chmod +x m3/big.bin && du -sh u38.0K u3Hard links are the other surprise. If two names point to the same file in the lower layer, writing through one of them only copies up that one, and the link is broken:
umount m3mkdir l4 u4 w4 m4 && echo one > l4/a && ln l4/a l4/bmount -t overlay overlay -o lowerdir=l4,upperdir=u4,workdir=w4 m4echo two > m4/a && cat m4/a m4/btwooneThe index=on option keeps track of copied-up files, and preserves hard
links:
umount m4 && rm -rf u4 w4 && mkdir u4 w4mount -t overlay overlay -o lowerdir=l4,upperdir=u4,workdir=w4,index=on m4echo two > m4/a && cat m4/a m4/btwotwoBoth options are off by default on this kernel. The defaults depend on how the
kernel was built, and can be checked in /sys/module/overlay/parameters/:
umount m4cat /sys/module/overlay/parameters/metacopy /sys/module/overlay/parameters/indexNNOverlays without root
Mounting an overlay used to require root. Since Linux 5.11, it’s also allowed inside a user namespace, which is what rootless container tools rely on. As a normal user, in a directory owned by that user:
mkdir lower upper work merged && echo "from lower" > lower/fmount -t overlay overlay -o lowerdir=lower,upperdir=upper,workdir=work mergedmount: /tmp/ovl-rootless/merged: must be superuser to use mount. dmesg(1) may have more information after failed mount system call.unshare --user --map-root-user --mount sh -c 'mount -t overlay overlay -o lowerdir=lower,upperdir=upper,workdir=work merged && cat merged/f'from lowerBefore that, rootless tools used fuse-overlayfs, which implements the same
thing in user space, on top of FUSE (see the post on FUSE).
Performance
To get an idea of the overhead, I compared random reads on files spread over 10 lower layers with the same reads on files in a normal directory:
mkdir upper work merged baselinefor i in {0..9}; do mkdir lower$i; head -c 1M /dev/urandom > lower$i/file$i; head -c 1M /dev/urandom > baseline/file$i; donemount -t overlay overlay -o lowerdir=$(ls -d lower* | paste -sd:),upperdir=upper,workdir=work mergedfio --name=randread --rw=randread --bs=4k --runtime=10s --time_based --group_reporting --numjobs=4 --filename=$(ls merged/file* | paste -sd:) --size=100%fio --name=randread --rw=randread --bs=4k --runtime=10s --time_based --group_reporting --numjobs=4 --filename=$(ls baseline/file* | paste -sd:) --size=100%On my personal laptop, with an SSD and ext4, the bandwidth was 486MB/s on the overlay mount and 588MB/s on the baseline directory, so an overhead of about 17%.
Lazy loading and the alternatives
With overlayfs, all the layers of an image have to be downloaded and unpacked before the container can start, which is what makes startup slow for large images. The alternatives I looked at:
- eStargz is an image format that lets the runtime fetch files on demand, instead of pulling whole layers first. It helps startup time, but it requires converting the existing images to this format.
- Nydus also loads images on demand, with its own image format and daemon. It is operationally complex, with many places where things can go wrong.
- fuse-overlayfs comes with a non-negligible performance cost, and file metadata and partition handling are tricky.
Additional resources
- The kernel documentation on overlayfs,
which covers all the mount options, including
metacopy,indexandredirect_dir. - The OCI image layer specification, for how whiteouts are stored in image layers.