debugging the kernel, or how to debug god

this guide is for linuxmaxxers, windowszzzzzzusers refrain!!!! get out!!!

userspace crashes and you get a stack trace y un raquetazzzzzzzzo. the kernel crashes and it takes the whole machine, your filesystem cache, and your last hour of unsaved notes down with it, then reboots like nothing happened. so rule zero of kernel work: you do NOT do it on the box you love. you do it on a disposable one that lives in a window you can close.

(you can do it on your own machine IF you're based enough ^-^)

why do this???

well, when you stop being a little bitch that only takes public PoCs from github and fires them at your clients, you'll start wondering "why does this work?", or "how can i make my own? find my own?". this little guide answers that.

and since the dawn of AI, finding CVEs has become what everybody does and the standard for greatness is so low in hell nothing matters anymore. you may as well point your agent at something that actually matters. like the linux kernel.

first you have to catch one

you cannot debug a kernel you don't have the source to. grab a release tarball from kernel.org, or clone the stable tree if you want history to blame later:

1
2

git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
cd linux

now the part everyone skips and then cries about: the config. a kernel is millions of lines behind thousands of CONFIG_ toggles, and a default build is tuned to be FAST and quiet, which is the exact opposite of what you want. start from a sane baseline, then flip on every snitch:

1
2
3
4
5
6
7
8

make defconfig
./scripts/config \
  -e DEBUG_INFO -e DEBUG_INFO_DWARF5 -e GDB_SCRIPTS \
  -e KALLSYMS -e KALLSYMS_ALL -e DEBUG_FS \
  -e KASAN -e KASAN_INLINE \
  -e LOCKDEP -e PROVE_LOCKING -e DEBUG_ATOMIC_SLEEP \
  -e FAULT_INJECTION -e FAILSLAB
make olddefconfig

quick rundown of what each flag actually does. DEBUG_INFO plus DWARF5 is what puts real symbols in vmlinux so gdb shows you start_kernel instead of a raw hex address. GDB_SCRIPTS gives you the lx- helper commands (list modules, dump dmesg from inside gdb, walk task lists). KASAN is the use-after-free tripwire we lean on in a minute, and KASAN_INLINE makes it faster so you'll actually leave it on. LOCKDEP and PROVE_LOCKING catch deadlocks BEFORE they hang the machine, by proving your lock ordering is sane. FAULT_INJECTION and FAILSLAB let you make allocations fail ON PURPOSE, which is how you test the error paths nobody ever tests. make olddefconfig at the end reconciles everything so the build doesn't stop to ask you 400 questions.

one shortcut if defconfig feels too bare: steal a config from a kernel that already works. every distro ships its own at /boot/config-$(uname -r), and if yours was built with CONFIG_IKCONFIG_PROC you can pull the running kernel's straight out of the live kernel with zcat /proc/config.gz > .config. copy one over, run make olddefconfig to fit it to your source version, then flip the same snitches on top. you get a config that definitely boots on real hardware, at the cost of a much fatter build.

if you're heading toward coverage-guided fuzzing later, add -e KCOV -d KCOV_INSTRUMENT_ALL -e KCOV_ENABLE_COMPARISONS while you're in here. syzkaller wants it and future-you will not want to rebuild.

then build it and go make coffee, because this is the slow part:

1

make -j$(nproc) bzImage

that spits out arch/x86/boot/bzImage (the compressed, bootable kernel) and, in the tree root, vmlinux (the fat one with all the symbols). qemu boots the first. gdb reads the second. do NOT mix them up and then wonder why every breakpoint is a lie.

a kernel with nothing to run is just a very fast panic

a kernel boots and then immediately looks for something to be pid 1. no init, no userspace, instant Kernel panic - not syncing: No working init found. so you hand it a tiny root filesystem in an initramfs (a cpio archive the kernel unpacks into a ramdisk). the lazy, self-contained way: build bash statically and make IT the init.

1
2
3
4
5
6
7
8
9
10

# static bash = zero shared libs to drag into the vm
cd bash-5.3 && ./configure --enable-static-link --without-bash-malloc && make -j$(nproc)

# lay out a rootfs and make bash pid 1
mkdir -p rootfs/{bin,dev,proc,sys}
cp bash rootfs/bin/ && ln -s bash rootfs/bin/sh
ln -s bin/bash rootfs/init          # the kernel runs /init

# pack it into the cpio qemu will boot
cd rootfs && find . | cpio -o -H newc > ../initrd.cpio

when bash comes up as pid 1 you get a shell and nothing else, no ps, no mounted /proc, so the first thing you type is mount -t proc none /proc && mount -t sysfs none /sys or half your tools lie to you. it's a feral little box, but it's YOURS, it boots in a second, and when you wedge the kernel you just kill qemu and boot it again. later you graduate to a real disk image (debian in a .img, ssh'd into over a hostfwd port) when you need actual packages. for learning, the cpio is enough.

two vms and a piece of string

the entire setup is qemu plus gdb talking over a socket. qemu boots your kernel, freezes before the first instruction, and waits for you to show up.

1
2
3
4
5

qemu-system-x86_64 \
  -kernel linux-6.17.7/arch/x86/boot/bzImage \
  -append "nokaslr console=ttyS0" \
  -initrd initrd.cpio \
  -m 2G -s -S -nographic

-s opens a gdb stub on port 1234. -S says freeze on boot and don't move until a debugger says so. nokaslr turns off address randomization, because you cannot set a breakpoint on a symbol that teleports somewhere new every boot. then in another terminal:

1
2
3
4

gdb linux-6.17.7/vmlinux
(gdb) target remote :1234
(gdb) b start_kernel
(gdb) c

that's it. that's the whole rig. you are now single-stepping the linux kernel on a machine you can delete. the vmlinux (not the compressed bzImage) is the one with the symbols, so gdb knows start_kernel from a hole in the ground.

hi im a war criminal

you learn a subsystem by breaking it on purpose, in the smallest possible way. so write a module that does exactly one illegal thing:

1
2
3
4
5
6
7
8
9

int use_after_free(void)
{
    char *leak = kzalloc(100, GFP_KERNEL);
    strcpy(leak, "this is a use-after-free example");
    kfree(leak);
    printk(KERN_INFO "using freed memory: %s\n", leak);
    return 0;
}
module_init(use_after_free);

allocate, free, then read the freed pointer anyway. in userspace this is a segfault if you're lucky and a silent wrong answer if you're not. in the kernel this is how CVEs are born. the memory is freed but the bytes are usually still sitting there, so it "works", right up until the allocator hands that slab to something else and now you're reading someone else's data. or writing it.

kasan is the narc you want

run that module on a normal kernel and it prints the string and shrugs. run it on a kernel built with CONFIG_KASAN=y and the machine loses its mind, in the good way:

1
2
3

BUG: KASAN: use-after-free in use_after_free+0x...
Read of size 33 at addr ffff8881...
Freed by task 1: kfree ... use_after_free

kasan is a shadow-memory tripwire the compiler wires into every load and store. it knows that address was freed four instructions ago and it will name the exact line, the exact task, and the exact size of your crime. this is the whole reason we flipped on all those debug toggles back at build time. you want the kernel to be a snitch. a quiet kernel is a kernel hiding bodies.

turning hex back into regret

most of the time you are NOT sitting in gdb. you've got a serial log with a splat in it, and the splat ends in something like use_after_free+0x1a/0x30. that's an offset into a function, not a line, and squinting at it accomplishes nothing. the kernel ships the translator:

1

./scripts/faddr2line vmlinux use_after_free+0x1a/0x30

out pops uaf.c:9, the exact line. (plain addr2line -e vmlinux does the same if you already have a raw address.) this unglamorous loop, read splat, faddr2line the top frame, go stare at that line, is where you actually spend your hours. single-stepping is for when faddr2line isn't enough.

and stay humble about it, because everything so far is EASY MODE. you built this kernel yourself, KASAN on, full symbols, /init you control. hunting a bug in someone else's kernel, a router, a phone, a locked appliance, you get NONE of that. no kasan holding your hand, symbols stripped, and the "splat" is a bare register dump if you get output at all. same skills, faddr2line against whatever symbols you can scrape back, but the training wheels are off. build with the snitch on to learn what a bug looks like. then go find it in the dark.

a module that doesn't commit crimes

destruction is a great teacher but eventually you want to build something that stays up. the smallest real thing is a character device: a node in /dev that answers read and write. here's a null device, /dev/null's dumber cousin.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

static struct file_operations my_null_fops = {
    .owner   = THIS_MODULE,
    .read    = my_null_read,
    .write   = my_null_write,
    .open    = my_null_open,
    .release = my_null_release,
};

ssize_t my_null_write(struct file *filp, const char __user *buf,
                      size_t count, loff_t *f_pos) {
    *f_pos += count;      // swallow it all, report it all written
    return count;
}

static int __init null_init(void) {
    dev_t devno = 0;
    alloc_chrdev_region(&devno, 0, 1, "my_null");   // let the kernel pick a major
    my_null_major = MAJOR(devno);
    cdev_init(&cdev, &my_null_fops);
    cdev_add(&cdev, devno, 1);
    return 0;
}

file_operations is the entire contract between userspace and your driver: fill in the callbacks you care about, leave the rest null (a null device leaving struct fields null, fittingly). read returns 0, which userspace reads as instant EOF. write eats every byte and lies that it wrote them all. and DON'T hardcode a major number, alloc_chrdev_region asks the kernel for a free one so you don't collide with a real driver; you read it back out of /proc/devices, mknod your node, and now cat, echo, open() all work against a thing you built. no crime. it just runs.

ftrace, or watching it think

sometimes nothing crashes and that's the problem. you want to know what path the kernel actually took, and ftrace draws you the whole call graph for free, no rebuild:

1
2
3
4

sudo trace-cmd record -p function_graph \
  -g "ieee80211_alloc_hw_nm" \
  /usr/bin/ping google.com -c1
trace-cmd report | less

function_graph gives you every function entered and left, indented like a flame graph made of text, with timings. point -g at one function and it traces the subtree under it. this is how you answer "wait, who is even CALLING this" without reading nine files of indirect pointers and losing your will to live.

trace-cmd record drops a trace.dat in the current directory, and report | less is fine until the trace is 40MB and you're scrolling through a text wall at 3am. so, for when you don't hate yourself: kernelshark trace.dat. it's the gui, same data, except now it's a zoomable timeline with every cpu and task laid out visually, click a spike to see the stack, filter by pid without memorizing flags. trace-cmd for the surgical one-liner, kernelshark for actually understanding what the machine did over time.

fuzzing is throwing rocks at god

once you can build, break, and watch, you automate the breaking. you fire malformed garbage at syscalls and wait for kasan to scream. the old-school one, trinity, opens its own README with a nuclear test quote and a warning that it "may seriously corrupt your files." it is not kidding. this is ALSO why you run it in the disposable vm and never, ever the host. the modern move is syzkaller, which is coverage-guided (it watches which kernel branches your inputs hit and steers toward the unexplored ones) and just does not get tired the way you do.

dirtycow, or how to milk root out of a race

time to point everything you just built at a legend. dirtycow (CVE-2016-5195) lived in the kernel for about NINE YEARS and it is the cleanest lesson in the genre, because the bug isn't a fancy overflow, it's the kernel being briefly indecisive about a copy.

the setup: you mmap a file you only have read access to, MAP_PRIVATE, so the deal is "you can scribble on this mapping, but the kernel makes you a private copy first (copy-on-write) and your writes go there, never the real file."

1
2

int fd = open("victim", O_RDONLY);        // root-owned, 0644, you can only READ it
void *map = mmap(NULL, 4096, PROT_READ, MAP_PRIVATE, fd, 0);

now you race two threads against each other. one keeps telling the kernel to throw your private copy away. the other keeps writing through /proc/self/mem, which forces the write down a path that skips the normal permission check:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

void *madvise_thread(void *_) {
    while (running)
        madvise(map, 4096, MADV_DONTNEED);    // "drop the private copy"
    return NULL;
}

void *write_thread(void *_) {
    int mem = open("/proc/self/mem", O_RDWR);
    while (running) {
        lseek(mem, (off_t)map, SEEK_SET);
        write(mem, "PWNED", 5);               // land this on the ORIGINAL page
    }
    return NULL;
}

the two weird primitives

if madvise and /proc/self/mem look like noise, here's what they actually are, because the race doesn't make sense until they do.

madvise(addr, len, advice) is you telling the kernel how you plan to use a chunk of memory so it can manage the pages smarter. it's ADVICE, not an order, but the kernel usually listens. MADV_DONTNEED is the spicy one: it means "i don't need these pages right now, throw them out." on a private mapping the kernel doesn't just forget them, it DISCARDS them, so the very next time you touch that address it has to fault a fresh page back in from scratch. that "fault it back in" is the whole lever. it lets you force the kernel to redo the page walk at a moment of your choosing.

/proc/self/mem is a file that literally IS your own address space. writing to it at an offset pokes that memory, but through the kernel's internal access path (the get_user_pages machinery) instead of a plain store. that path is the one with the buggy copy-on-write handling, and it's happy to write to a page that a normal PROT_READ mapping would have slapped you for.

now the race ties together. the /proc/self/mem write starts a copy-on-write: kernel decides to make your private copy, and there's a gap between "decided" and "committed". MADV_DONTNEED, fired a few thousand times a second, aims to land inside that gap and delete the copy before it's committed. win the gap and the write retries against the ORIGINAL page. lose it and nothing happens, so you just spin until you win. a race condition is exactly this: two operations whose result depends on which one gets there first, and here you get unlimited retries to rig the coin.

here's the whole bug in one breath. the write faults in a copy-on-write page, the kernel goes "ok, making your private copy now", and in the sliver of time before it actually commits that copy, the other thread's MADV_DONTNEED yanks it. the kernel retries, re-walks to the page, and gets the ORIGINAL read-only file page back, except it's already convinced this write was allowed. so your bytes land in the page cache of a file you were NEVER allowed to write. point victim at /etc/passwd and you rewrite root's line. the "cow" is copy-on-write; the "dirty" is that you dirtied a page you had no business touching.

that's it. no shellcode, no ROP, no leak. a scheduling window the width of a hair, hit a few thousand times a second until you win. build an old enough kernel in your throwaway vm (it's patched now, obviously) and you can watch a read-only file change under your own hands. THAT is the moment kernel bugs stop being abstract.

the evidence locker

a use-after-free in the right subsystem is how a phone gets rooted from a text message, how a locked device in an evidence locker gets opened, how a box running a hospital's records gets owned before anyone notices. the bytes you free and read back for fun are, somewhere else, the layer under someone's whole life. that's the reason the good ones report it and the reason the ceiling on this skill is so high. the kernel is the floor everything else stands on, and the floor does not get a second chance to validate its input.

anyway. build the snitch kernel, write the module that commits the crime, and let kasan tattle. the machine reboots. you don't lose anything you weren't ready to lose.

the wizard walks by, casting its shadow, weaving its spell!!!!! :3