1Written by: Neil Brown 2Please see MAINTAINERS file for where to send questions. 3 4Overlay Filesystem 5================== 6 7This document describes a prototype for a new approach to providing 8overlay-filesystem functionality in Linux (sometimes referred to as 9union-filesystems). An overlay-filesystem tries to present a 10filesystem which is the result over overlaying one filesystem on top 11of the other. 12 13The result will inevitably fail to look exactly like a normal 14filesystem for various technical reasons. The expectation is that 15many use cases will be able to ignore these differences. 16 17This approach is 'hybrid' because the objects that appear in the 18filesystem do not all appear to belong to that filesystem. In many 19cases an object accessed in the union will be indistinguishable 20from accessing the corresponding object from the original filesystem. 21This is most obvious from the 'st_dev' field returned by stat(2). 22 23While directories will report an st_dev from the overlay-filesystem, 24all non-directory objects will report an st_dev from the lower or 25upper filesystem that is providing the object. Similarly st_ino will 26only be unique when combined with st_dev, and both of these can change 27over the lifetime of a non-directory object. Many applications and 28tools ignore these values and will not be affected. 29 30Upper and Lower 31--------------- 32 33An overlay filesystem combines two filesystems - an 'upper' filesystem 34and a 'lower' filesystem. When a name exists in both filesystems, the 35object in the 'upper' filesystem is visible while the object in the 36'lower' filesystem is either hidden or, in the case of directories, 37merged with the 'upper' object. 38 39It would be more correct to refer to an upper and lower 'directory 40tree' rather than 'filesystem' as it is quite possible for both 41directory trees to be in the same filesystem and there is no 42requirement that the root of a filesystem be given for either upper or 43lower. 44 45The lower filesystem can be any filesystem supported by Linux and does 46not need to be writable. The lower filesystem can even be another 47overlayfs. The upper filesystem will normally be writable and if it 48is it must support the creation of trusted.* extended attributes, and 49must provide valid d_type in readdir responses, so NFS is not suitable. 50 51A read-only overlay of two read-only filesystems may use any 52filesystem type. 53 54Directories 55----------- 56 57Overlaying mainly involves directories. If a given name appears in both 58upper and lower filesystems and refers to a non-directory in either, 59then the lower object is hidden - the name refers only to the upper 60object. 61 62Where both upper and lower objects are directories, a merged directory 63is formed. 64 65At mount time, the two directories given as mount options "lowerdir" and 66"upperdir" are combined into a merged directory: 67 68 mount -t overlay overlay -olowerdir=/lower,upperdir=/upper,\ 69workdir=/work /merged 70 71The "workdir" needs to be an empty directory on the same filesystem 72as upperdir. 73 74Then whenever a lookup is requested in such a merged directory, the 75lookup is performed in each actual directory and the combined result 76is cached in the dentry belonging to the overlay filesystem. If both 77actual lookups find directories, both are stored and a merged 78directory is created, otherwise only one is stored: the upper if it 79exists, else the lower. 80 81Only the lists of names from directories are merged. Other content 82such as metadata and extended attributes are reported for the upper 83directory only. These attributes of the lower directory are hidden. 84 85whiteouts and opaque directories 86-------------------------------- 87 88In order to support rm and rmdir without changing the lower 89filesystem, an overlay filesystem needs to record in the upper filesystem 90that files have been removed. This is done using whiteouts and opaque 91directories (non-directories are always opaque). 92 93A whiteout is created as a character device with 0/0 device number. 94When a whiteout is found in the upper level of a merged directory, any 95matching name in the lower level is ignored, and the whiteout itself 96is also hidden. 97 98A directory is made opaque by setting the xattr "trusted.overlay.opaque" 99to "y". Where the upper filesystem contains an opaque directory, any 100directory in the lower filesystem with the same name is ignored. 101 102readdir 103------- 104 105When a 'readdir' request is made on a merged directory, the upper and 106lower directories are each read and the name lists merged in the 107obvious way (upper is read first, then lower - entries that already 108exist are not re-added). This merged name list is cached in the 109'struct file' and so remains as long as the file is kept open. If the 110directory is opened and read by two processes at the same time, they 111will each have separate caches. A seekdir to the start of the 112directory (offset 0) followed by a readdir will cause the cache to be 113discarded and rebuilt. 114 115This means that changes to the merged directory do not appear while a 116directory is being read. This is unlikely to be noticed by many 117programs. 118 119seek offsets are assigned sequentially when the directories are read. 120Thus if 121 - read part of a directory 122 - remember an offset, and close the directory 123 - re-open the directory some time later 124 - seek to the remembered offset 125 126there may be little correlation between the old and new locations in 127the list of filenames, particularly if anything has changed in the 128directory. 129 130Readdir on directories that are not merged is simply handled by the 131underlying directory (upper or lower). 132 133 134Non-directories 135--------------- 136 137Objects that are not directories (files, symlinks, device-special 138files etc.) are presented either from the upper or lower filesystem as 139appropriate. When a file in the lower filesystem is accessed in a way 140the requires write-access, such as opening for write access, changing 141some metadata etc., the file is first copied from the lower filesystem 142to the upper filesystem (copy_up). Note that creating a hard-link 143also requires copy_up, though of course creation of a symlink does 144not. 145 146The copy_up may turn out to be unnecessary, for example if the file is 147opened for read-write but the data is not modified. 148 149The copy_up process first makes sure that the containing directory 150exists in the upper filesystem - creating it and any parents as 151necessary. It then creates the object with the same metadata (owner, 152mode, mtime, symlink-target etc.) and then if the object is a file, the 153data is copied from the lower to the upper filesystem. Finally any 154extended attributes are copied up. 155 156Once the copy_up is complete, the overlay filesystem simply 157provides direct access to the newly created file in the upper 158filesystem - future operations on the file are barely noticed by the 159overlay filesystem (though an operation on the name of the file such as 160rename or unlink will of course be noticed and handled). 161 162 163Multiple lower layers 164--------------------- 165 166Multiple lower layers can now be given using the the colon (":") as a 167separator character between the directory names. For example: 168 169 mount -t overlay overlay -olowerdir=/lower1:/lower2:/lower3 /merged 170 171As the example shows, "upperdir=" and "workdir=" may be omitted. In 172that case the overlay will be read-only. 173 174The specified lower directories will be stacked beginning from the 175rightmost one and going left. In the above example lower1 will be the 176top, lower2 the middle and lower3 the bottom layer. 177 178 179Non-standard behavior 180--------------------- 181 182The copy_up operation essentially creates a new, identical file and 183moves it over to the old name. The new file may be on a different 184filesystem, so both st_dev and st_ino of the file may change. 185 186Any open files referring to this inode will access the old data and 187metadata. Similarly any file locks obtained before copy_up will not 188apply to the copied up file. 189 190On a file opened with O_RDONLY fchmod(2), fchown(2), futimesat(2) and 191fsetxattr(2) will fail with EROFS. 192 193If a file with multiple hard links is copied up, then this will 194"break" the link. Changes will not be propagated to other names 195referring to the same inode. 196 197Symlinks in /proc/PID/ and /proc/PID/fd which point to a non-directory 198object in overlayfs will not contain valid absolute paths, only 199relative paths leading up to the filesystem's root. This will be 200fixed in the future. 201 202Some operations are not atomic, for example a crash during copy_up or 203rename will leave the filesystem in an inconsistent state. This will 204be addressed in the future. 205 206Changes to underlying filesystems 207--------------------------------- 208 209Offline changes, when the overlay is not mounted, are allowed to either 210the upper or the lower trees. 211 212Changes to the underlying filesystems while part of a mounted overlay 213filesystem are not allowed. If the underlying filesystem is changed, 214the behavior of the overlay is undefined, though it will not result in 215a crash or deadlock. 216 217Testsuite 218--------- 219 220There's testsuite developed by David Howells at: 221 222 git://git.infradead.org/users/dhowells/unionmount-testsuite.git 223 224Run as root: 225 226 # cd unionmount-testsuite 227 # ./run --ov 228