1Introduction 2============ 3 4dm-cache is a device mapper target written by Joe Thornber, Heinz 5Mauelshagen, and Mike Snitzer. 6 7It aims to improve performance of a block device (eg, a spindle) by 8dynamically migrating some of its data to a faster, smaller device 9(eg, an SSD). 10 11This device-mapper solution allows us to insert this caching at 12different levels of the dm stack, for instance above the data device for 13a thin-provisioning pool. Caching solutions that are integrated more 14closely with the virtual memory system should give better performance. 15 16The target reuses the metadata library used in the thin-provisioning 17library. 18 19The decision as to what data to migrate and when is left to a plug-in 20policy module. Several of these have been written as we experiment, 21and we hope other people will contribute others for specific io 22scenarios (eg. a vm image server). 23 24Glossary 25======== 26 27 Migration - Movement of the primary copy of a logical block from one 28 device to the other. 29 Promotion - Migration from slow device to fast device. 30 Demotion - Migration from fast device to slow device. 31 32The origin device always contains a copy of the logical block, which 33may be out of date or kept in sync with the copy on the cache device 34(depending on policy). 35 36Design 37====== 38 39Sub-devices 40----------- 41 42The target is constructed by passing three devices to it (along with 43other parameters detailed later): 44 451. An origin device - the big, slow one. 46 472. A cache device - the small, fast one. 48 493. A small metadata device - records which blocks are in the cache, 50 which are dirty, and extra hints for use by the policy object. 51 This information could be put on the cache device, but having it 52 separate allows the volume manager to configure it differently, 53 e.g. as a mirror for extra robustness. This metadata device may only 54 be used by a single cache device. 55 56Fixed block size 57---------------- 58 59The origin is divided up into blocks of a fixed size. This block size 60is configurable when you first create the cache. Typically we've been 61using block sizes of 256KB - 1024KB. The block size must be between 64 62(32KB) and 2097152 (1GB) and a multiple of 64 (32KB). 63 64Having a fixed block size simplifies the target a lot. But it is 65something of a compromise. For instance, a small part of a block may be 66getting hit a lot, yet the whole block will be promoted to the cache. 67So large block sizes are bad because they waste cache space. And small 68block sizes are bad because they increase the amount of metadata (both 69in core and on disk). 70 71Cache operating modes 72--------------------- 73 74The cache has three operating modes: writeback, writethrough and 75passthrough. 76 77If writeback, the default, is selected then a write to a block that is 78cached will go only to the cache and the block will be marked dirty in 79the metadata. 80 81If writethrough is selected then a write to a cached block will not 82complete until it has hit both the origin and cache devices. Clean 83blocks should remain clean. 84 85If passthrough is selected, useful when the cache contents are not known 86to be coherent with the origin device, then all reads are served from 87the origin device (all reads miss the cache) and all writes are 88forwarded to the origin device; additionally, write hits cause cache 89block invalidates. To enable passthrough mode the cache must be clean. 90Passthrough mode allows a cache device to be activated without having to 91worry about coherency. Coherency that exists is maintained, although 92the cache will gradually cool as writes take place. If the coherency of 93the cache can later be verified, or established through use of the 94"invalidate_cblocks" message, the cache device can be transitioned to 95writethrough or writeback mode while still warm. Otherwise, the cache 96contents can be discarded prior to transitioning to the desired 97operating mode. 98 99A simple cleaner policy is provided, which will clean (write back) all 100dirty blocks in a cache. Useful for decommissioning a cache or when 101shrinking a cache. Shrinking the cache's fast device requires all cache 102blocks, in the area of the cache being removed, to be clean. If the 103area being removed from the cache still contains dirty blocks the resize 104will fail. Care must be taken to never reduce the volume used for the 105cache's fast device until the cache is clean. This is of particular 106importance if writeback mode is used. Writethrough and passthrough 107modes already maintain a clean cache. Future support to partially clean 108the cache, above a specified threshold, will allow for keeping the cache 109warm and in writeback mode during resize. 110 111Migration throttling 112-------------------- 113 114Migrating data between the origin and cache device uses bandwidth. 115The user can set a throttle to prevent more than a certain amount of 116migration occurring at any one time. Currently we're not taking any 117account of normal io traffic going to the devices. More work needs 118doing here to avoid migrating during those peak io moments. 119 120For the time being, a message "migration_threshold <#sectors>" 121can be used to set the maximum number of sectors being migrated, 122the default being 204800 sectors (or 100MB). 123 124Updating on-disk metadata 125------------------------- 126 127On-disk metadata is committed every time a FLUSH or FUA bio is written. 128If no such requests are made then commits will occur every second. This 129means the cache behaves like a physical disk that has a volatile write 130cache. If power is lost you may lose some recent writes. The metadata 131should always be consistent in spite of any crash. 132 133The 'dirty' state for a cache block changes far too frequently for us 134to keep updating it on the fly. So we treat it as a hint. In normal 135operation it will be written when the dm device is suspended. If the 136system crashes all cache blocks will be assumed dirty when restarted. 137 138Per-block policy hints 139---------------------- 140 141Policy plug-ins can store a chunk of data per cache block. It's up to 142the policy how big this chunk is, but it should be kept small. Like the 143dirty flags this data is lost if there's a crash so a safe fallback 144value should always be possible. 145 146For instance, the 'mq' policy, which is currently the default policy, 147uses this facility to store the hit count of the cache blocks. If 148there's a crash this information will be lost, which means the cache 149may be less efficient until those hit counts are regenerated. 150 151Policy hints affect performance, not correctness. 152 153Policy messaging 154---------------- 155 156Policies will have different tunables, specific to each one, so we 157need a generic way of getting and setting these. Device-mapper 158messages are used. Refer to cache-policies.txt. 159 160Discard bitset resolution 161------------------------- 162 163We can avoid copying data during migration if we know the block has 164been discarded. A prime example of this is when mkfs discards the 165whole block device. We store a bitset tracking the discard state of 166blocks. However, we allow this bitset to have a different block size 167from the cache blocks. This is because we need to track the discard 168state for all of the origin device (compare with the dirty bitset 169which is just for the smaller cache device). 170 171Target interface 172================ 173 174Constructor 175----------- 176 177 cache <metadata dev> <cache dev> <origin dev> <block size> 178 <#feature args> [<feature arg>]* 179 <policy> <#policy args> [policy args]* 180 181 metadata dev : fast device holding the persistent metadata 182 cache dev : fast device holding cached data blocks 183 origin dev : slow device holding original data blocks 184 block size : cache unit size in sectors 185 186 #feature args : number of feature arguments passed 187 feature args : writethrough or passthrough (The default is writeback.) 188 189 policy : the replacement policy to use 190 #policy args : an even number of arguments corresponding to 191 key/value pairs passed to the policy 192 policy args : key/value pairs passed to the policy 193 E.g. 'sequential_threshold 1024' 194 See cache-policies.txt for details. 195 196Optional feature arguments are: 197 writethrough : write through caching that prohibits cache block 198 content from being different from origin block content. 199 Without this argument, the default behaviour is to write 200 back cache block contents later for performance reasons, 201 so they may differ from the corresponding origin blocks. 202 203 passthrough : a degraded mode useful for various cache coherency 204 situations (e.g., rolling back snapshots of 205 underlying storage). Reads and writes always go to 206 the origin. If a write goes to a cached origin 207 block, then the cache block is invalidated. 208 To enable passthrough mode the cache must be clean. 209 210A policy called 'default' is always registered. This is an alias for 211the policy we currently think is giving best all round performance. 212 213As the default policy could vary between kernels, if you are relying on 214the characteristics of a specific policy, always request it by name. 215 216Status 217------ 218 219<metadata block size> <#used metadata blocks>/<#total metadata blocks> 220<cache block size> <#used cache blocks>/<#total cache blocks> 221<#read hits> <#read misses> <#write hits> <#write misses> 222<#demotions> <#promotions> <#dirty> <#features> <features>* 223<#core args> <core args>* <policy name> <#policy args> <policy args>* 224<cache metadata mode> 225 226metadata block size : Fixed block size for each metadata block in 227 sectors 228#used metadata blocks : Number of metadata blocks used 229#total metadata blocks : Total number of metadata blocks 230cache block size : Configurable block size for the cache device 231 in sectors 232#used cache blocks : Number of blocks resident in the cache 233#total cache blocks : Total number of cache blocks 234#read hits : Number of times a READ bio has been mapped 235 to the cache 236#read misses : Number of times a READ bio has been mapped 237 to the origin 238#write hits : Number of times a WRITE bio has been mapped 239 to the cache 240#write misses : Number of times a WRITE bio has been 241 mapped to the origin 242#demotions : Number of times a block has been removed 243 from the cache 244#promotions : Number of times a block has been moved to 245 the cache 246#dirty : Number of blocks in the cache that differ 247 from the origin 248#feature args : Number of feature args to follow 249feature args : 'writethrough' (optional) 250#core args : Number of core arguments (must be even) 251core args : Key/value pairs for tuning the core 252 e.g. migration_threshold 253policy name : Name of the policy 254#policy args : Number of policy arguments to follow (must be even) 255policy args : Key/value pairs e.g. sequential_threshold 256cache metadata mode : ro if read-only, rw if read-write 257 In serious cases where even a read-only mode is deemed unsafe 258 no further I/O will be permitted and the status will just 259 contain the string 'Fail'. The userspace recovery tools 260 should then be used. 261needs_check : 'needs_check' if set, '-' if not set 262 A metadata operation has failed, resulting in the needs_check 263 flag being set in the metadata's superblock. The metadata 264 device must be deactivated and checked/repaired before the 265 cache can be made fully operational again. '-' indicates 266 needs_check is not set. 267 268Messages 269-------- 270 271Policies will have different tunables, specific to each one, so we 272need a generic way of getting and setting these. Device-mapper 273messages are used. (A sysfs interface would also be possible.) 274 275The message format is: 276 277 <key> <value> 278 279E.g. 280 dmsetup message my_cache 0 sequential_threshold 1024 281 282 283Invalidation is removing an entry from the cache without writing it 284back. Cache blocks can be invalidated via the invalidate_cblocks 285message, which takes an arbitrary number of cblock ranges. Each cblock 286range's end value is "one past the end", meaning 5-10 expresses a range 287of values from 5 to 9. Each cblock must be expressed as a decimal 288value, in the future a variant message that takes cblock ranges 289expressed in hexidecimal may be needed to better support efficient 290invalidation of larger caches. The cache must be in passthrough mode 291when invalidate_cblocks is used. 292 293 invalidate_cblocks [<cblock>|<cblock begin>-<cblock end>]* 294 295E.g. 296 dmsetup message my_cache 0 invalidate_cblocks 2345 3456-4567 5678-6789 297 298Examples 299======== 300 301The test suite can be found here: 302 303https://github.com/jthornber/device-mapper-test-suite 304 305dmsetup create my_cache --table '0 41943040 cache /dev/mapper/metadata \ 306 /dev/mapper/ssd /dev/mapper/origin 512 1 writeback default 0' 307dmsetup create my_cache --table '0 41943040 cache /dev/mapper/metadata \ 308 /dev/mapper/ssd /dev/mapper/origin 1024 1 writeback \ 309 mq 4 sequential_threshold 1024 random_threshold 8' 310