Demystifying Overlay File Systems

Posted May 11, 2025

By InfluentCoder

22 min read

Introduction

Overlay file systems are one of those powerful but often misunderstood features of the Linux kernel. If you’ve ever worked with containers, used a live Linux distribution, or built sandboxed environments, chances are you’ve already relied on an overlay file system—even if you didn’t realize it.

At its core, an overlay file system lets you stack multiple directories together so they appear as a single unified file system. One or more of these layers can be read-only, while the topmost layer is writable. This gives you the illusion of modifying a file system without actually changing the original data.

Visualizing Overlay Layers

Here’s a simplified diagram to illustrate the basic structure of an overlay file system:

Overlay Mount Point
│
└── Merged View (what you see)
    ├── From upperdir (writable)
    └── From lowerdir (read-only)

┌────────────────────────────────────────────────────────┐
│ upperdir/ │ ← Writable layer (e.g., container changes) │
└────────────────────────────────────────────────────────┘
     ▲
     │
┌────────────────────────────────────────────────────────┐
│ lowerdir/ │ ← Read-only layer (e.g., base image)       │
└────────────────────────────────────────────────────────┘

This layering allows changes to be stored in upperdir while keeping the lowerdir untouched. Deletions, additions, and modifications are all handled by the overlay mechanism transparently.

Quick Example: Manually Creating an Overlay Mount

Let’s look at a simple example using the mount command. This shows how to set up an overlay file system manually:

  
mkdir -p /tmp/overlay/{lower,upper,work,merged}

# Create a file in the lower directory (simulating a base image)
echo "from base layer" > /tmp/overlay/lower/hello.txt

# Mount the overlay
sudo mount -t overlay overlay -o lowerdir=/tmp/overlay/lower,upperdir=/tmp/overlay/upper,workdir=/tmp/overlay/work /tmp/overlay/merged

# Inspect the merged view
cat /tmp/overlay/merged/hello.txt
# Output: from base layer

# Modify the file in the merged view
echo "modified in overlay" > /tmp/overlay/merged/hello.txt

# The original lower file is untouched
cat /tmp/overlay/lower/hello.txt
# Output: from base layer

# The modified version is visible from the merged view
cat /tmp/overlay/merged/hello.txt
# Output: modified in overlay

# The modified version is written to the upper layer
cat /tmp/overlay/upper/hello.txt
# Output: modified in overlay

# Clean up
sudo umount /tmp/overlay/merged

This example shows the copy-on-write behavior in action. The original hello.txt in the lower layer remains unchanged, while the modified version is stored in the upper layer and presented through the merged view.

Overlay file systems are the secret sauce behind container runtimes like Docker and Podman. They enable copy-on-write semantics, allowing containers to start quickly, share common base images, and isolate changes made at runtime. But they’re also useful outside of containers—in development environments, testing setups, and even backup systems.

In this article, we’ll explore how overlay file systems work under the hood, why they’re used so widely in modern infrastructure, and what you need to watch out for when using them. Whether you’re an infrastructure engineer trying to debug container behavior, or a curious developer wanting to learn more about Linux internals, this guide aims to give you a clear and practical understanding of overlay file systems.

The Concept of File System Layers

At the heart of an overlay file system is a simple but powerful idea: stacking multiple directories to create a unified view. This is especially useful when you want to combine read-only data with a writable workspace, while keeping both logically separate.

Imagine you’re working with a base directory that contains system files or a container image layer. You want to make changes—like modifying files or installing packages—but you don’t want to touch the original files. Overlay file systems make this possible.

Read-Only and Writable Layers

Overlay file systems typically involve two main types of layers:

Lower Layer(s): These are read-only directories. You can stack multiple lower layers (e.g., image layers in Docker), but they can’t be modified directly.
Upper Layer: This is the writable directory. Any changes you make—creating, modifying, or deleting files—are recorded here.
Work Directory: A scratch space required by the overlay kernel module to manage writes (especially for complex operations like renames).

When mounted, these layers are merged into a virtual view. Reads pull from the upper layer if the file exists there, otherwise from the lower layer. Writes always go to the upper layer.

A Layered Analogy

Think of this like layers in Photoshop:

The bottom layers (lowerdir) contain your background image and base artwork.
The top layer (upperdir) is where you paint or add temporary stickers.
When you save your work, only the top layer changes—the base image remains intact.

This layered approach enables features like:

Isolation: You can experiment without modifying the original files.
Efficiency: Common base layers can be shared across multiple environments.
Rollback: Discard changes by removing the writable layer.

Example: What Happens When You Modify a File

Let’s say you’re using an overlay file system with:

lowerdir: contains /etc/config
upperdir: initially empty

Now you run:

  
echo "new config" > /merged/etc/config

Here’s what happens:

The overlay system sees that /etc/config exists in lowerdir but not in upperdir.
It copies up the file from lowerdir to upperdir.
Your write is applied to the copy in upperdir.
From that point on, any read of /etc/config comes from upperdir.

This copy-on-write behavior is what makes overlays so powerful for containers and other isolated environments.

In the next section, we’ll take a closer look at how Linux implements this functionality through overlayfs, and how you can use it directly on your system.

A Gentle Dive into `overlayfs` (the Linux Implementation)

The Linux kernel’s implementation of an overlay file system is called overlayfs. Introduced in kernel 3.18 and stabilized over time, it’s the default choice for container runtimes like Docker, Podman, and containerd.

Unlike user-space solutions like UnionFS or AUFS, overlayfs is implemented entirely in the kernel, which means it’s fast, simple, and widely available in modern distributions.

Key Concepts in `overlayfs`

overlayfs merges two (or more) directories into a single mount point. To do this, it requires three key components:

lowerdir: The read-only base layer (e.g., a container image layer).
upperdir: The writable layer where changes are stored.
workdir: A scratch space used internally by the kernel to support operations like file renames.

You mount them like this:

  
sudo mount -t overlay overlay -o lowerdir=/path/to/lower,upperdir=/path/to/upper,workdir=/path/to/work /path/to/merged

Once mounted, any interaction with /path/to/merged is transparently handled by overlayfs.

How Reads and Writes Work

Here’s the general rule:

Reads: If a file exists in the upperdir, it’s read from there. Otherwise, it falls back to the lowerdir.
Writes: Always go to the upperdir. If the file doesn’t exist in upperdir but does in lowerdir, it is copied up before being modified.

Let’s break it down with a table:

Operation	Behavior
Read file	Read from upperdir if present, else lowerdir
Write file	Copy-up from lowerdir to upperdir, then modify
Create new file	Created directly in upperdir
Delete file	“Whiteout” file is created in upperdir to mask lower file

What Are Whiteouts?

In overlayfs, whiteouts are special marker files that hide files from the lowerdir. If you delete a file that exists in the lowerdir, overlayfs creates a whiteout in the upperdir.

This means the file still exists in the lowerdir, but the overlay system hides it from the merged view:

  
rm /merged/foo.txt
ls -l /upperdir
# c--------- 1 root root 0, 0 ... foo.txt

This character device file tells the overlay mount logic:

“Hide foo.txt from lowerdir in the merged view.”

So while the file still physically exists in lowerdir, it will no longer appear in /merged.

Note: Older union filesystems like AUFS used a .wh.foo.txt naming convention for whiteouts, but overlayfs relies on the character device approach for kernel-level handling.

Real-World Use Case: Containers

Overlay file systems are one of the key building blocks behind modern container runtimes like Docker, Podman, and containerd. These tools rely on overlay layering to make containers fast, efficient, and isolated.

Let’s unpack how overlayfs is used in this context.

Image Layers and Copy-on-Write

When you pull a container image, you’re not getting a single monolithic file. Instead, the image is composed of multiple read-only layers, each representing a filesystem diff (new files, changed files, etc.).

When you start a container from an image:

The image layers become the lowerdir.
A new, empty upperdir is created for your container’s writable layer.
The runtime mounts them using overlayfs to produce a merged view.

This design means:

All containers based on the same image share the same base layers.
Each container only stores its own changes.
Deleting a container doesn’t affect the image or other containers.

Container Mount (merged view)
       │
       ├── upperdir: container changes (writable)
       └── lowerdir: base image layers (read-only)

Example: How Docker Uses OverlayFS

Docker uses overlay2 as the default storage driver, which is a user-space integration that leverages the Linux kernel’s overlayfs. It supports multiple lower layers and is more flexible than the older overlay driver used in earlier Docker versions.

You can inspect this by running:

docker info | grep Storage
# Storage Driver: overlay2

Each image layer is stored in /var/lib/docker/overlay2/<layer-id>/diff, and the runtime sets up the correct lower and upper directories for each container mount.

Here’s a simplified example of what happens when you run:

docker run -it ubuntu bash

Docker prepares the base Ubuntu image layers (read-only).
It creates a writable layer for the container.
It mounts all of them using overlayfs:

  
sudo mount -t overlay overlay -o \
  lowerdir=layer1:layer2:layer3,\
  upperdir=/var/lib/docker/overlay2/container-id/diff,\
  workdir=/var/lib/docker/overlay2/container-id/work \
  /var/lib/docker/overlay2/container-id/merged

The container process is launched in a chroot/jail rooted at the merged directory.

You can actually cd into /var/lib/docker/overlay2/... to explore these layers if you’re curious.

Performance and Disk Efficiency

This model is incredibly efficient for containerized environments:

Speed: Starting a new container is instant—it doesn’t need to copy a whole root filesystem.
Storage: Multiple containers share the same image layers, minimizing disk usage.
Isolation: Each container sees its own filesystem changes without affecting others.

Container snapshots, commits, and image builds are all built around the idea of stacking and managing these layers.

In the next section, we’ll look at the limitations and edge cases you might encounter when working with overlayfs — because while powerful, it’s not without quirks.

Limitations and Gotchas

While overlayfs is powerful and widely used, it comes with a set of limitations and quirks that can surprise you — especially if you’re using it outside of Docker’s happy path or in custom infrastructure setups.

Here are some important caveats to be aware of:

Filesystem Compatibility

Not all filesystems are created equal when it comes to overlayfs.

lowerdir and upperdir must reside on the same filesystem type.
Most commonly supported: ext4, xfs (with f_type enabled), and btrfs.
NFS is not supported as an upperdir (and often fails even as lowerdir).
tmpfs can be used in upperdir for ephemeral, memory-backed overlays.

Hard Link Limitations

Hard links do not work across layers.

If a file in lowerdir is hard-linked, those links aren’t preserved when the file is copied to upperdir.

Some tools (e.g., package managers) rely on hard links and may behave differently under overlayfs.

Metadata-Only Changes

Changing just a file’s metadata (e.g., permissions, timestamps, ownership) causes a copy-up of the entire file—even though content remains the same.

This can result in unexpected performance and storage usage.

chmod +x /merged/bin/tool
# This causes a full copy of the binary into upperdir

Lower Layer Ordering Matters

If you’re stacking multiple lower layers (supported on newer kernels), the order in which you list them matters.
Only the first occurrence of a file is visible in the merged view.

  
sudo mount -t overlay overlay -o lowerdir=layerA:layerB,...
# If both have /etc/foo.conf, only the one in layerA is seen

SELinux and Mount Contexts

When SELinux is enabled, overlayfs mounts can be blocked or behave unexpectedly unless explicitly allowed.

You may need to use the context= mount option or disable SELinux for testing (obviously not recommended).

  
sudo mount -t overlay overlay -o lowerdir=...,context="system_u:object_r:container_file_t:s0" ...

Troubleshooting Can Be Tricky

You can’t easily tell whether a file is coming from upperdir or lowerdir just by looking at /merged.
Tools like debugfs, stat, and checking inode numbers can help — but this adds complexity.
- But even looking at inodes can be tricky, as overlayfs may not always show the expected inode numbers. If you look at the above example where we do a metadata-only change, the inode number of the file in upperdir may match the original inode number in lowerdir. However, this is in reality a different file with a different inode.
Deleting a file doesn’t physically remove it — it creates a whiteout in the upper layer, which may be confusing when debugging file presence.

In the next section, we’ll look at alternatives and enhancements to overlayfs, including options for rootless containers, user-space overlays, and lazy-loading image formats.

Overlay Alternatives and Enhancements

While overlayfs is the most common layered file system used in Linux containers today, it’s not the only option. Depending on your use case—especially in rootless environments, advanced image delivery, or multi-user setups—other solutions might be better suited.

UnionFS, AUFS, and MergerFS

These were the early attempts at union file systems on Linux. While they’re mostly obsolete in container ecosystems, they introduced key ideas that overlayfs built upon.

UnionFS

One of the first implementations of layered filesystems.
Fully in user space.
Largely unmaintained today.

AUFS (Advanced Union File System)

Used by Docker before overlayfs was stable.
Supports advanced features (e.g., per-file whiteouts, inode merging).
No longer included in mainline Linux.
Requires out-of-tree kernel patches.

MergerFS

User-space union filesystem with FUSE - still under active development as of this writing.
Popular for home NAS setups, media servers, and JBOD storage pools.
Not suitable for container workloads (no copy-on-write, no upperdir semantics).

🧑‍💻 Rootless Containers: `fuse-overlayfs`

Standard overlayfs requires root privileges because it depends on kernel namespaces and mounting capabilities. This is a problem for rootless containers, which are increasingly common for improved security and multi-tenant systems.

`fuse-overlayfs`

A user-space implementation of overlayfs using FUSE.
Used by Podman and Buildah for rootless containers.
Supports multiple lower layers and copy-on-write behavior.
Performance is decent, but slower than in-kernel overlayfs.

  
# Example: rootless Podman using fuse-overlayfs
podman info | grep overlay
# overlay driver: fuse-overlayfs

Lazy-Loaded Overlay Systems

To improve container startup times and reduce bandwidth usage, some systems introduce lazy loading and on-demand fetch of image layers.

eStargz
- An image format designed for lazy loading.
- Files are fetched as they are accessed.
- Developed by Google/Containerd ecosystem.
Nydus
- More advanced than eStargz.
- Uses a user-space FUSE daemon with a custom format.
- Focused on cloud-native workloads with high image churn.
Stargz Snapshotter
- A containerd plugin that uses eStargz format.
- Fetches image layers over the network as needed.
- Great for large images with cold-start concerns.

When Should You Consider Alternatives?

Use Case	Recommended Option
Rootless container builds	`fuse-overlayfs`
Read-only media libraries	`mergerfs`
Speeding up large container pulls	`eStargz` or `Nydus`
Legacy Docker support	AUFS (legacy only)
General-purpose containers	`overlayfs` (kernel)

In the next section, we’ll examine how performance is affected by overlay stacking, what operations are slow, and how to optimize your overlay-based workflows.

Performance Considerations

overlayfs is efficient for most use cases—but like all abstractions, it comes with trade-offs. Understanding how reads, writes, and inode lookups behave can help you diagnose slow containers, bloated layers, or unexpected I/O overhead.

Read vs Write Performance

Reads

Fast, especially when files are only in the lowerdir.
When a file is present in the upperdir, reads go there directly.
If you’re stacking many lowerdirs, reads may degrade slightly due to lookup order.

Writes

Slower than native filesystems due to copy-up on first write.
Writing to a file that exists only in the lower layer triggers:
1. Copy of the file into the upperdir.
2. Modification of the copy.

This means the first write is expensive — after that, it’s as fast as a normal write on the underlying FS.

  
# This triggers a full copy-up:
echo "change" > /merged/usr/bin/tool

Inode and Metadata Overhead

Every file lookup involves checking the upperdir, then falling back to the lowerdir(s).
Overlayfs stores merged metadata in memory, but still does inode lookups per access.
Stat-heavy operations (like find, ls -lR, or some build tools) may perform worse than expected.

Layer Depth Matters

Stacking many layers can slow down lookup times:

  
mount -t overlay overlay -o \
  lowerdir=layer10:layer9:...:layer1,...

In this case:

For each path lookup, the kernel walks through each lowerdir until it finds the first matching file.
This affects container runtimes with many image layers (e.g., micro-layers from multi-stage builds).

To mitigate:

Use multi-layer squashing during image builds (e.g., docker export, buildah, or skopeo).
Minimize the number of layers for performance-critical containers.

Measuring Overlay Performance

You can use standard Linux tools to benchmark overlay performance:

fio for read/write I/O patterns
time, strace, perf for syscall-level behavior
du -s and ls -i to inspect copy-up and inode reuse

Example:

  
# Create 10 files of 1MB each in 10 separate lower directories
mkdir upper work merged
for i in {0..10}; do mkdir lower${i}; head -c 1M /dev/urandom > lower${i}/file${i}; done
sudo mount -t overlay overlay -o lowerdir=`python3 -c 'print(":".join([f"lower{i}" for i in range(10)]))'`,upperdir=upper,workdir=work merged
# Baseline: create 10 files of 1MB each in a single directory
mkdir baseline
for i in {0..10}; do head -c 1M /dev/urandom > baseline/file${i}; done
# Run test on the overlay mount
fio --name=randread --rw=randread --bs=4k --runtime=10s --time_based --group_reporting --numjobs=4 --filename=`python3 -c 'print(":".join([f"merged/file{i}" for i in range(10)]))'` --size=100%
# Run test on the baseline directory
fio --name=randread --rw=randread --bs=4k --runtime=10s --time_based --group_reporting --numjobs=4 --filename=`python3 -c 'print(":".join([f"baseline/file{i}" for i in range(10)]))'` --size=100%
# Compare the results, bandwidth, latency etc.

On my personal laptop using a SSD and ext4 filesystem, the bandwidth on the overlay mount is 486MB/s, whereas it is 588MB/s on the baseline directory, so the overhead is ~17%.

Optimization Tips

Place upperdir and workdir on fast local storage (SSD or tmpfs).
Avoid running overlayfs over NFS or other networked filesystems.
Use lazy-loading formats (like eStargz) for large container images with low locality.
Be aware of performance traps in CI pipelines that reuse overlay mounts heavily.

In the next section, we’ll walk through practical debugging and troubleshooting techniques—because overlay mounts can be tricky to inspect once they’re live.

Debugging and Troubleshooting Overlay Mounts

Working with overlay file systems can be tricky because the merged view hides which layer a file is actually coming from. When something behaves unexpectedly—like a missing file, failed write, or inconsistent state—you’ll need to dig beneath the surface.

Here are practical tips and techniques for troubleshooting overlay mounts.

Determining File Origin

You can’t tell at a glance whether a file comes from upperdir or lowerdir. But here are some ways to find out:

Check inodes

If the same file exists in both layers, their inodes will differ:

  
stat /merged/foo.txt
stat /upper/foo.txt
stat /lower/foo.txt

The one in /merged will match /upper only if it was copied up.

Compare file contents

diff /merged/foo.txt /lower/foo.txt
# If different, the file was copied up and modified
# Note that this is not foolproof, as the file may have been copied up due to a metadata change like permissions

Understanding Whiteouts

If a file exists in lowerdir but is missing in the merged view, it might be whiteouted.

Check upperdir for character devices with the same filename:

  
find /upper -type c -ls
# Look for files with major/minor 0:0

If you find:

c--------- 1 root root 0, 0 ... foo.txt

That means the file was deleted (whiteouted) at some point.

Interpreting Common Errors

Object is remote

This usually means you’re using an NFS or network FS that doesn’t fully support overlayfs.

Invalid argument

Likely causes:

Mixing incompatible filesystems (e.g., lowerdir on NFS, upperdir on ext4)
Workdir not on same fs as upperdir
Typo in mount options

Useful Commands and Tools

Inspect mounts

mount | grep overlay

Check overlayfs status in /proc

  
cat /proc/mounts | grep overlay
cat /proc/self/mountinfo | grep overlay

Debug overlay file locations

  
ls -l /upper/
ls -l /work/
ls -l /lower/
ls -l /merged/

Compare entries, timestamps, and inodes across these layers.

Cleaning Up

Overlay mounts don’t always clean up after themselves—especially after failed container runs.

Always unmount before deleting:

umount /merged

Then delete upperdir, workdir, and merged mountpoints if needed.

rm -rf /upper /work /merged

Failure to unmount can lead to “device or resource busy” errors.

In the next section, we’ll go beyond container use cases and explore how overlayfs can be used for snapshotting, sandboxing, and layered development environments.

Advanced Use Cases

While overlayfs is most famously used in containers, its power extends far beyond. If you’re building systems that require layering, sandboxing, or fast, reversible changes, overlayfs gives you low-level tools to do it right.

Here are some advanced use cases you may not have considered:

Sandboxed Development Environments

Want to let developers experiment freely without breaking a base system? Use an overlay mount with a read-only lowerdir and a writable tmpfs upperdir.

  
mount -t overlay overlay -o \
  lowerdir=/opt/devtools,\
  upperdir=/dev/shm/dev-sandbox,\
  workdir=/dev/shm/work \
  /mnt/devbox

Any changes are non-persistent.
Restarting clears the sandbox.
Great for demo environments, tutorials, or student workspaces.

CI/CD Testing and Isolation

In CI systems, tests often install dependencies, mutate filesystems, or run in parallel.

Overlay-based approaches let you:

Run tests on a shared base image layer
Discard mutations after test run
Isolate jobs using tmpfs-based overlays

Bonus: this can be faster than full container launches in lightweight environments.

Safe System Upgrades or Patching

You can overlay a patched version of /usr or /lib onto a system, test it live, and roll back by unmounting the overlay.

  
mount -t overlay overlay -o \
  lowerdir=/usr,\
  upperdir=/opt/upgrade/usr,\
  workdir=/opt/upgrade/work \
  /mnt/test-usr

Test new binaries or libraries without risk.
Use with chroot, systemd-nspawn, or custom init environments.

Snapshots and Rollbacks

Combine overlayfs with a snapshot-friendly FS (like btrfs or LVM) to create checkpointable environments.

Workflow:

Base system → lowerdir
Snapshot current state → new upperdir
Mount overlay → perform changes
Roll back by discarding upperdir

Some systems (like Apptainer/Singularity) do something similar for scientific workloads.

Container Layer Authoring and Debugging

Building container images by hand? Overlayfs is perfect for iterating without rebuilding full images.

Start with image contents as lowerdir
Mount with an empty upperdir
Make changes
Inspect upperdir for diff
Package that into a new image layer

This gives you precise control over layer contents and avoids slow rebuilds during debugging.

In the final section, we’ll wrap up what we’ve learned, summarize key trade-offs, and talk about when you should reach for overlayfs—and when you shouldn’t.

Closing Thoughts

Overlay file systems are a powerful example of how simple ideas—like layering directories—can unlock massive efficiencies in software deployment, isolation, and reproducibility. Whether you’re working with containers, building sandbox environments, or managing system upgrades, overlayfs gives you the building blocks to do it with elegance and speed.

Recap: What Makes OverlayFS Great

Copy-on-write behavior allows lightweight isolation.
Layer stacking promotes reusability and disk efficiency.
Performance is solid for most workloads—especially read-heavy ones.
Container-native: It’s the engine beneath Docker, Podman, and containerd.
Flexible: You can use it outside containers to build sandboxes, test harnesses, or rollback systems.

But It’s Not Perfect

Doesn’t play well with NFS or other network filesystems.
Has quirks around metadata changes, whiteouts, and hard links.
Debugging overlay behavior can require deep filesystem knowledge.
Performance can degrade with deep layer stacks or heavy copy-up.

When Should You Use OverlayFS?

Use it when:

You need fast, isolated environments (e.g., containers, dev sandboxes).
You want to experiment without affecting the base system.
You’re optimizing for space by reusing base layers.

Avoid it when:

You need full filesystem fidelity (e.g., hard link preservation, NFS compatibility).
You’re doing write-heavy workloads that frequently copy large files.
You need consistent behavior across all Linux distributions or filesystems.

Final Thoughts

The best tools in infrastructure are the ones that disappear into the background—but empower everything. OverlayFS is exactly that kind of tool. It’s everywhere in modern computing, from containers to live systems, and understanding it unlocks a deeper mastery of Linux and system design.

Thanks for reading! If you found this useful or want to share your own overlay use case, feel free to drop a comment or reach out.

Want to dig deeper? Check out the kernel docs on overlayfs or experiment with your own layered mount setup.

linux

This post is licensed under CC BY 4.0 by the author.