Linux Kernel Debugging.pdf

Linux Technology Center Linux Kernel Debugging Oops, Now What? Ross Mikosh James Washer Linux Technology Center Types of Problems Tools Error and Debug Messages Handling Failures Kernel Investigation Handling a System Crash Oops Analysis Example LKCD/Lcrash More Information Outline Linux Technology Center Debuggers gdb kdb others? Built-In Oops data upon a panic/crash Dump Facility Linux? Kernel Crash Dump - lkcd Linux Trace Toolkit ltt Custom Kernel Instrumentation dprobes Special console functions Magic SysReq key Tools Linux Technology Center System error logs /var/log/* dmesg Syslog Console Application or Module debug level Error/Debug Messages Linux Technology Center System Crash Collect and analyze oops/panic data Collect and analyze dump with lkcd System Hang Use Magic SysReq key NMI invoking a dump using lkcd S/390 - invoke a stand-alone dump Handling Failures Linux Technology Center Debuggers Gdb and /proc/kcore Remote kernel debugging - gdb serial connection Kdb Lcrash on running system Adding printks in the kernel Programming a debug module or a new /proc file Appropriate for customer/production environment ? Kernel Investigation Linux Technology Center Occurs when a critical system failure is detected Kernel routines call die() Attempts to report/record system state Information is limited Better to have an entire system memory dump LKCD project on SourceForge Thorough analysis and investigation can be done Handling a System Crash Linux Technology Center Panic/Oops Analysis Steps Collect oops output, System.map, /proc/ksyms, vmlinux, /proc/modules Use ksymoops to interpret oops Instructions is /usr/src/linux/Documentation/oops-tracing.txt Ksymoops(8) man page Be Careful... Brief analysis Ksymoops disassembles the code section The EIP points to the failing instruction The call trace section shows how you got there Caution: Noise on the stack? How to find failing line of code? Oops Example Unable to handle kernel NULL pointer dereference at virtual ad