1Fault injection capabilities infrastructure 2=========================================== 3 4See also drivers/md/faulty.c and "every_nth" module option for scsi_debug. 5 6 7Available fault injection capabilities 8-------------------------------------- 9 10o failslab 11 12 injects slab allocation failures. (kmalloc(), kmem_cache_alloc(), ...) 13 14o fail_page_alloc 15 16 injects page allocation failures. (alloc_pages(), get_free_pages(), ...) 17 18o fail_make_request 19 20 injects disk IO errors on devices permitted by setting 21 /sys/block/<device>/make-it-fail or 22 /sys/block/<device>/<partition>/make-it-fail. (generic_make_request()) 23 24o fail_mmc_request 25 26 injects MMC data errors on devices permitted by setting 27 debugfs entries under /sys/kernel/debug/mmc0/fail_mmc_request 28 29Configure fault-injection capabilities behavior 30----------------------------------------------- 31 32o debugfs entries 33 34fault-inject-debugfs kernel module provides some debugfs entries for runtime 35configuration of fault-injection capabilities. 36 37- /sys/kernel/debug/fail*/probability: 38 39 likelihood of failure injection, in percent. 40 Format: <percent> 41 42 Note that one-failure-per-hundred is a very high error rate 43 for some testcases. Consider setting probability=100 and configure 44 /sys/kernel/debug/fail*/interval for such testcases. 45 46- /sys/kernel/debug/fail*/interval: 47 48 specifies the interval between failures, for calls to 49 should_fail() that pass all the other tests. 50 51 Note that if you enable this, by setting interval>1, you will 52 probably want to set probability=100. 53 54- /sys/kernel/debug/fail*/times: 55 56 specifies how many times failures may happen at most. 57 A value of -1 means "no limit". 58 59- /sys/kernel/debug/fail*/space: 60 61 specifies an initial resource "budget", decremented by "size" 62 on each call to should_fail(,size). Failure injection is 63 suppressed until "space" reaches zero. 64 65- /sys/kernel/debug/fail*/verbose 66 67 Format: { 0 | 1 | 2 } 68 specifies the verbosity of the messages when failure is 69 injected. '0' means no messages; '1' will print only a single 70 log line per failure; '2' will print a call trace too -- useful 71 to debug the problems revealed by fault injection. 72 73- /sys/kernel/debug/fail*/task-filter: 74 75 Format: { 'Y' | 'N' } 76 A value of 'N' disables filtering by process (default). 77 Any positive value limits failures to only processes indicated by 78 /proc/<pid>/make-it-fail==1. 79 80- /sys/kernel/debug/fail*/require-start: 81- /sys/kernel/debug/fail*/require-end: 82- /sys/kernel/debug/fail*/reject-start: 83- /sys/kernel/debug/fail*/reject-end: 84 85 specifies the range of virtual addresses tested during 86 stacktrace walking. Failure is injected only if some caller 87 in the walked stacktrace lies within the required range, and 88 none lies within the rejected range. 89 Default required range is [0,ULONG_MAX) (whole of virtual address space). 90 Default rejected range is [0,0). 91 92- /sys/kernel/debug/fail*/stacktrace-depth: 93 94 specifies the maximum stacktrace depth walked during search 95 for a caller within [require-start,require-end) OR 96 [reject-start,reject-end). 97 98- /sys/kernel/debug/fail_page_alloc/ignore-gfp-highmem: 99 100 Format: { 'Y' | 'N' } 101 default is 'N', setting it to 'Y' won't inject failures into 102 highmem/user allocations. 103 104- /sys/kernel/debug/failslab/ignore-gfp-wait: 105- /sys/kernel/debug/fail_page_alloc/ignore-gfp-wait: 106 107 Format: { 'Y' | 'N' } 108 default is 'N', setting it to 'Y' will inject failures 109 only into non-sleep allocations (GFP_ATOMIC allocations). 110 111- /sys/kernel/debug/fail_page_alloc/min-order: 112 113 specifies the minimum page allocation order to be injected 114 failures. 115 116o Boot option 117 118In order to inject faults while debugfs is not available (early boot time), 119use the boot option: 120 121 failslab= 122 fail_page_alloc= 123 fail_make_request= 124 mmc_core.fail_request=<interval>,<probability>,<space>,<times> 125 126How to add new fault injection capability 127----------------------------------------- 128 129o #include <linux/fault-inject.h> 130 131o define the fault attributes 132 133 DECLARE_FAULT_INJECTION(name); 134 135 Please see the definition of struct fault_attr in fault-inject.h 136 for details. 137 138o provide a way to configure fault attributes 139 140- boot option 141 142 If you need to enable the fault injection capability from boot time, you can 143 provide boot option to configure it. There is a helper function for it: 144 145 setup_fault_attr(attr, str); 146 147- debugfs entries 148 149 failslab, fail_page_alloc, and fail_make_request use this way. 150 Helper functions: 151 152 fault_create_debugfs_attr(name, parent, attr); 153 154- module parameters 155 156 If the scope of the fault injection capability is limited to a 157 single kernel module, it is better to provide module parameters to 158 configure the fault attributes. 159 160o add a hook to insert failures 161 162 Upon should_fail() returning true, client code should inject a failure. 163 164 should_fail(attr, size); 165 166Application Examples 167-------------------- 168 169o Inject slab allocation failures into module init/exit code 170 171#!/bin/bash 172 173FAILTYPE=failslab 174echo Y > /sys/kernel/debug/$FAILTYPE/task-filter 175echo 10 > /sys/kernel/debug/$FAILTYPE/probability 176echo 100 > /sys/kernel/debug/$FAILTYPE/interval 177echo -1 > /sys/kernel/debug/$FAILTYPE/times 178echo 0 > /sys/kernel/debug/$FAILTYPE/space 179echo 2 > /sys/kernel/debug/$FAILTYPE/verbose 180echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait 181 182faulty_system() 183{ 184 bash -c "echo 1 > /proc/self/make-it-fail && exec $*" 185} 186 187if [ $# -eq 0 ] 188then 189 echo "Usage: $0 modulename [ modulename ... ]" 190 exit 1 191fi 192 193for m in $* 194do 195 echo inserting $m... 196 faulty_system modprobe $m 197 198 echo removing $m... 199 faulty_system modprobe -r $m 200done 201 202------------------------------------------------------------------------------ 203 204o Inject page allocation failures only for a specific module 205 206#!/bin/bash 207 208FAILTYPE=fail_page_alloc 209module=$1 210 211if [ -z $module ] 212then 213 echo "Usage: $0 <modulename>" 214 exit 1 215fi 216 217modprobe $module 218 219if [ ! -d /sys/module/$module/sections ] 220then 221 echo Module $module is not loaded 222 exit 1 223fi 224 225cat /sys/module/$module/sections/.text > /sys/kernel/debug/$FAILTYPE/require-start 226cat /sys/module/$module/sections/.data > /sys/kernel/debug/$FAILTYPE/require-end 227 228echo N > /sys/kernel/debug/$FAILTYPE/task-filter 229echo 10 > /sys/kernel/debug/$FAILTYPE/probability 230echo 100 > /sys/kernel/debug/$FAILTYPE/interval 231echo -1 > /sys/kernel/debug/$FAILTYPE/times 232echo 0 > /sys/kernel/debug/$FAILTYPE/space 233echo 2 > /sys/kernel/debug/$FAILTYPE/verbose 234echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait 235echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-highmem 236echo 10 > /sys/kernel/debug/$FAILTYPE/stacktrace-depth 237 238trap "echo 0 > /sys/kernel/debug/$FAILTYPE/probability" SIGINT SIGTERM EXIT 239 240echo "Injecting errors into the module $module... (interrupt to stop)" 241sleep 1000000 242 243Tool to run command with failslab or fail_page_alloc 244---------------------------------------------------- 245In order to make it easier to accomplish the tasks mentioned above, we can use 246tools/testing/fault-injection/failcmd.sh. Please run a command 247"./tools/testing/fault-injection/failcmd.sh --help" for more information and 248see the following examples. 249 250Examples: 251 252Run a command "make -C tools/testing/selftests/ run_tests" with injecting slab 253allocation failure. 254 255 # ./tools/testing/fault-injection/failcmd.sh \ 256 -- make -C tools/testing/selftests/ run_tests 257 258Same as above except to specify 100 times failures at most instead of one time 259at most by default. 260 261 # ./tools/testing/fault-injection/failcmd.sh --times=100 \ 262 -- make -C tools/testing/selftests/ run_tests 263 264Same as above except to inject page allocation failure instead of slab 265allocation failure. 266 267 # env FAILCMD_TYPE=fail_page_alloc \ 268 ./tools/testing/fault-injection/failcmd.sh --times=100 \ 269 -- make -C tools/testing/selftests/ run_tests 270