Mount namespaces and mount propagation in Linux
/ 7 min read
Table of Contents
At work, I was working on a compute platform that schedules workloads on shared hardware. We don’t run them in containers, so we do the isolation ourselves. I also worked on supporting containers for short workloads, where fast container startup matters, and I looked into things like lazy loading of images (eStargz) and fuse-overlayfs.
This was a new area for me. I had never really had to dig into how containers work, and mounts turned out to be a big part of it: what a process sees as its filesystem is decided by its mount namespace, and by how mounts propagate between namespaces.
This post goes through both, with experiments you can run on any Linux machine. Overlay filesystems and FUSE have their own posts: overlay file systems and FUSE.
Mounts and the mount table
Linux has a single directory tree, and a mount attaches a filesystem at some
directory of that tree (the mount point). The list of mounts a process sees is
its mount table, which findmnt shows as a tree, and /proc/self/mountinfo
shows in raw form.
The important part is that this mount table is not global. It belongs to a mount namespace, and every process is in one.
Setting up the experiments
Everything below runs as root, in two terminals. The first one plays the role of the host. Let’s create a small tmpfs to play with, so we don’t touch any real mount:
mkdir -p /mnt/demo && mount -t tmpfs demo /mnt/demo && mount --make-shared /mnt/demofindmnt -o TARGET,FSTYPE,PROPAGATION /mnt/demoTARGET FSTYPE PROPAGATION/mnt/demo tmpfs sharedIgnore --make-shared for now, we’ll come back to it.
The outputs below are from a real run. Namespace IDs and mount IDs will be different on your machine.
A new mount namespace
In the second terminal, let’s start a shell in a new mount namespace:
unshare --mount bashThe two shells are now in different namespaces, which /proc/self/ns/mnt
shows. In terminal 1:
readlink /proc/self/ns/mntmnt:[4026532264]And in terminal 2:
readlink /proc/self/ns/mntmnt:[4026532265]The new namespace starts with a copy of the mount table, so /mnt/demo is
there. Let’s mount something under it, in terminal 2:
mkdir /mnt/demo/from-t2 && mount -t tmpfs t2 /mnt/demo/from-t2 && echo hello > /mnt/demo/from-t2/filefindmnt -o TARGET,FSTYPE /mnt/demo/from-t2; cat /mnt/demo/from-t2/fileTARGET FSTYPE/mnt/demo/from-t2 tmpfshelloAnd look for it from terminal 1:
ls /mnt/demo; findmnt /mnt/demo/from-t2; echo "exit code: $?"; ls /mnt/demo/from-t2from-t2exit code: 1The from-t2 directory is visible,
because it was created on the /mnt/demo tmpfs, which both namespaces use.
But the mount on top of it only exists in terminal 2’s namespace, so from
terminal 1 the directory is empty. Namespaces isolate mounts, not files.
Why the mount didn’t leak: propagation
Remember the --make-shared? In terminal 1, /mnt/demo is shared, which
means mount events under it should propagate to its copies. Yet nothing
propagated. In terminal 2, findmnt shows why:
findmnt -o TARGET,FSTYPE,PROPAGATION /mnt/demoTARGET FSTYPE PROPAGATION/mnt/demo tmpfs privateunshare (since util-linux 2.27) recursively makes every mount private in the
new namespace, so that it’s really isolated. This is usually what you want,
but it hides how propagation works. Let’s exit the shell in terminal 2 and
start a new one, keeping the propagation as it is:
unshare --mount --propagation unchanged bashfindmnt -o TARGET,FSTYPE,PROPAGATION /mnt/demoTARGET FSTYPE PROPAGATION/mnt/demo tmpfs sharedShared mounts
Now /mnt/demo is shared in both namespaces. Mounting something under it in
terminal 2:
mkdir /mnt/demo/from-t2 && mount -t tmpfs t2 /mnt/demo/from-t2shows up in terminal 1:
findmnt -o TARGET,FSTYPE,PROPAGATION /mnt/demo/from-t2TARGET FSTYPE PROPAGATION/mnt/demo/from-t2 tmpfs sharedAnd it works the other way too. In terminal 1:
mkdir /mnt/demo/from-t1 && mount -t tmpfs t1 /mnt/demo/from-t1In terminal 2:
findmnt -o TARGET,FSTYPE,PROPAGATION /mnt/demo/from-t1TARGET FSTYPE PROPAGATION/mnt/demo/from-t1 tmpfs sharedWhat links the two copies of /mnt/demo is a peer group. It’s visible in
/proc/self/mountinfo as shared:N. In terminal 1:
grep /mnt/demo /proc/self/mountinfo76 50 0:41 / /mnt/demo rw,relatime shared:1 - tmpfs demo rw106 76 0:42 / /mnt/demo/from-t2 rw,relatime shared:2 - tmpfs t2 rw107 76 0:43 / /mnt/demo/from-t1 rw,relatime shared:3 - tmpfs t1 rwAnd in terminal 2:
grep /mnt/demo /proc/self/mountinfo104 78 0:41 / /mnt/demo rw,relatime shared:1 - tmpfs demo rw105 104 0:42 / /mnt/demo/from-t2 rw,relatime shared:2 - tmpfs t2 rw108 104 0:43 / /mnt/demo/from-t1 rw,relatime shared:3 - tmpfs t1 rwThe mount IDs (first column) are different, since these are different mounts
in different namespaces, but they are in the same peer groups (shared:1,
shared:2, shared:3). A mount or unmount under one member of a peer group
is replayed under all the others.
Slave mounts
Often, we want something in between: mounts made on the host should appear in the namespace, but not the other way around. That’s a slave mount. In terminal 2:
mount --make-slave /mnt/demo && findmnt -o TARGET,PROPAGATION /mnt/demoTARGET PROPAGATION/mnt/demo private,slaveA new mount in terminal 1:
mkdir /mnt/demo/from-t1-again && mount -t tmpfs t1b /mnt/demo/from-t1-againstill shows up in terminal 2:
findmnt -o TARGET,FSTYPE /mnt/demo/from-t1-againTARGET FSTYPE/mnt/demo/from-t1-again tmpfsBut a mount made in terminal 2:
mkdir /mnt/demo/from-t2-again && mount -t tmpfs t2b /mnt/demo/from-t2-againdoesn’t show up in terminal 1:
findmnt /mnt/demo/from-t2-again; echo "exit code: $?"exit code: 1In mountinfo, the mount in terminal 2 now has master:1 instead of
shared:1: it receives events from peer group 1, but isn’t part of it.
grep /mnt/demo /proc/self/mountinfo | head -1104 78 0:41 / /mnt/demo rw,relatime master:1 - tmpfs demo rwThe propagation modes
So there are four modes:
shared: mount events propagate both ways, between all the members of the peer group.slave: events from the master’s peer group propagate in, nothing propagates out. A mount can be both slave and shared, it then forwards what it receives to its own peers.private: nothing propagates in or out. This is the kernel default for new mounts.unbindable: private, and it also can’t be the source of a bind mount.
mount --make-shared (and the others) change a single mount. The recursive
versions, --make-rshared, --make-rslave, --make-rprivate and
--make-runbindable, change the mount and all the mounts under it, which is
usually what you want for a whole tree like /.
To clean up, exit the shell in terminal 2 (which removes its namespace and its mounts), then in terminal 1:
umount -R /mnt/demoWhat this means for containers
A container runtime does the same thing as unshare: it creates a mount
namespace, makes the mounts in it private or slave, then mounts the image’s
root filesystem and switches to it. So by default, mounts made on the host
after the container started don’t show up inside, and mounts made inside
don’t leak to the host.
Bind mounts from the host follow the same rules. Docker’s
bind-propagation option defaults to rprivate. With rslave, mounts that
appear later on the host, under the bind-mounted directory, also appear in the
container. rshared also sends the container’s mounts back to the host, and
only works if the source directory is itself a shared mount on the host.
Without a container runtime, the same building blocks are available directly:
unshare, or the unshare(2) and setns(2) system calls, plus mount and
the propagation flags.
Inspecting mount namespaces
The commands used above, plus a few more:
findmnt -o TARGET,PROPAGATIONshows the propagation of each mount./proc/<pid>/mountinfoshows the mount table of any process, with theshared:Nandmaster:Ntags.readlink /proc/<pid>/ns/mntshows which mount namespace a process is in. Two processes with the same value see the same mounts.lsns -t mntlists the mount namespaces on the machine, and a process in each.nsenter --mount=/proc/<pid>/ns/mntruns a command in the mount namespace of another process, for example to check what a container really sees.
Additional resources
- Shared subtrees, the kernel documentation on propagation, with many more examples.
- mount_namespaces(7), which covers the rules in detail, including what happens with user namespaces.