1Debugging kernel and modules via gdb 2==================================== 3 4The kernel debugger kgdb, hypervisors like QEMU or JTAG-based hardware 5interfaces allow to debug the Linux kernel and its modules during runtime 6using gdb. Gdb comes with a powerful scripting interface for python. The 7kernel provides a collection of helper scripts that can simplify typical 8kernel debugging steps. This is a short tutorial about how to enable and use 9them. It focuses on QEMU/KVM virtual machines as target, but the examples can 10be transferred to the other gdb stubs as well. 11 12 13Requirements 14------------ 15 16 o gdb 7.2+ (recommended: 7.4+) with python support enabled (typically true 17 for distributions) 18 19 20Setup 21----- 22 23 o Create a virtual Linux machine for QEMU/KVM (see www.linux-kvm.org and 24 www.qemu.org for more details). For cross-development, 25 http://landley.net/aboriginal/bin keeps a pool of machine images and 26 toolchains that can be helpful to start from. 27 28 o Build the kernel with CONFIG_GDB_SCRIPTS enabled, but leave 29 CONFIG_DEBUG_INFO_REDUCED off. If your architecture supports 30 CONFIG_FRAME_POINTER, keep it enabled. 31 32 o Install that kernel on the guest. 33 34 Alternatively, QEMU allows to boot the kernel directly using -kernel, 35 -append, -initrd command line switches. This is generally only useful if 36 you do not depend on modules. See QEMU documentation for more details on 37 this mode. 38 39 o Enable the gdb stub of QEMU/KVM, either 40 - at VM startup time by appending "-s" to the QEMU command line 41 or 42 - during runtime by issuing "gdbserver" from the QEMU monitor 43 console 44 45 o cd /path/to/linux-build 46 47 o Start gdb: gdb vmlinux 48 49 Note: Some distros may restrict auto-loading of gdb scripts to known safe 50 directories. In case gdb reports to refuse loading vmlinux-gdb.py, add 51 52 add-auto-load-safe-path /path/to/linux-build 53 54 to ~/.gdbinit. See gdb help for more details. 55 56 o Attach to the booted guest: 57 (gdb) target remote :1234 58 59 60Examples of using the Linux-provided gdb helpers 61------------------------------------------------ 62 63 o Load module (and main kernel) symbols: 64 (gdb) lx-symbols 65 loading vmlinux 66 scanning for modules in /home/user/linux/build 67 loading @0xffffffffa0020000: /home/user/linux/build/net/netfilter/xt_tcpudp.ko 68 loading @0xffffffffa0016000: /home/user/linux/build/net/netfilter/xt_pkttype.ko 69 loading @0xffffffffa0002000: /home/user/linux/build/net/netfilter/xt_limit.ko 70 loading @0xffffffffa00ca000: /home/user/linux/build/net/packet/af_packet.ko 71 loading @0xffffffffa003c000: /home/user/linux/build/fs/fuse/fuse.ko 72 ... 73 loading @0xffffffffa0000000: /home/user/linux/build/drivers/ata/ata_generic.ko 74 75 o Set a breakpoint on some not yet loaded module function, e.g.: 76 (gdb) b btrfs_init_sysfs 77 Function "btrfs_init_sysfs" not defined. 78 Make breakpoint pending on future shared library load? (y or [n]) y 79 Breakpoint 1 (btrfs_init_sysfs) pending. 80 81 o Continue the target 82 (gdb) c 83 84 o Load the module on the target and watch the symbols being loaded as well as 85 the breakpoint hit: 86 loading @0xffffffffa0034000: /home/user/linux/build/lib/libcrc32c.ko 87 loading @0xffffffffa0050000: /home/user/linux/build/lib/lzo/lzo_compress.ko 88 loading @0xffffffffa006e000: /home/user/linux/build/lib/zlib_deflate/zlib_deflate.ko 89 loading @0xffffffffa01b1000: /home/user/linux/build/fs/btrfs/btrfs.ko 90 91 Breakpoint 1, btrfs_init_sysfs () at /home/user/linux/fs/btrfs/sysfs.c:36 92 36 btrfs_kset = kset_create_and_add("btrfs", NULL, fs_kobj); 93 94 o Dump the log buffer of the target kernel: 95 (gdb) lx-dmesg 96 [ 0.000000] Initializing cgroup subsys cpuset 97 [ 0.000000] Initializing cgroup subsys cpu 98 [ 0.000000] Linux version 3.8.0-rc4-dbg+ (... 99 [ 0.000000] Command line: root=/dev/sda2 resume=/dev/sda1 vga=0x314 100 [ 0.000000] e820: BIOS-provided physical RAM map: 101 [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable 102 [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved 103 .... 104 105 o Examine fields of the current task struct: 106 (gdb) p $lx_current().pid 107 $1 = 4998 108 (gdb) p $lx_current().comm 109 $2 = "modprobe\000\000\000\000\000\000\000" 110 111 o Make use of the per-cpu function for the current or a specified CPU: 112 (gdb) p $lx_per_cpu("runqueues").nr_running 113 $3 = 1 114 (gdb) p $lx_per_cpu("runqueues", 2).nr_running 115 $4 = 0 116 117 o Dig into hrtimers using the container_of helper: 118 (gdb) set $next = $lx_per_cpu("hrtimer_bases").clock_base[0].active.next 119 (gdb) p *$container_of($next, "struct hrtimer", "node") 120 $5 = { 121 node = { 122 node = { 123 __rb_parent_color = 18446612133355256072, 124 rb_right = 0x0 <irq_stack_union>, 125 rb_left = 0x0 <irq_stack_union> 126 }, 127 expires = { 128 tv64 = 1835268000000 129 } 130 }, 131 _softexpires = { 132 tv64 = 1835268000000 133 }, 134 function = 0xffffffff81078232 <tick_sched_timer>, 135 base = 0xffff88003fd0d6f0, 136 state = 1, 137 start_pid = 0, 138 start_site = 0xffffffff81055c1f <hrtimer_start_range_ns+20>, 139 start_comm = "swapper/2\000\000\000\000\000\000" 140 } 141 142 143List of commands and functions 144------------------------------ 145 146The number of commands and convenience functions may evolve over the time, 147this is just a snapshot of the initial version: 148 149 (gdb) apropos lx 150 function lx_current -- Return current task 151 function lx_module -- Find module by name and return the module variable 152 function lx_per_cpu -- Return per-cpu variable 153 function lx_task_by_pid -- Find Linux task by PID and return the task_struct variable 154 function lx_thread_info -- Calculate Linux thread_info from task variable 155 lx-dmesg -- Print Linux kernel log buffer 156 lx-lsmod -- List currently loaded modules 157 lx-symbols -- (Re-)load symbols of Linux kernel and currently loaded modules 158 159Detailed help can be obtained via "help <command-name>" for commands and "help 160function <function-name>" for convenience functions. 161