Linux Kernel Module Debugging – GDB setup

I will explain how to debug my own loadable kernel module with remote debugging.

First we need to tell the gdb we are using remote debugging, in case of Ethernet we need to set the debugee IP and port (i.e. target remote localhost:8832), If we are using serial , this will be the commands:
(gdb)set remotebaud 115200
(gdb)target remote /dev/ttyS0

I will use the “hello world” module we have created(hello-world-kernel-module) for this demonstration.

We need to install the compiled module in our guest machine, so copy it to the machine and type:
# insmod hello.ko
(We can verify the insmod by seeking the output in the “/var/log/messages/ file, please refer to the “hello world” post).

Now, we need to add our module's symbols into the debugger, we will need to know the address of our module after we load it into the kernel.
The module start address can be found using the following command on our guest machine (after the insmod):
# cat /sys/module/hello/sections/.text
0xd0b1a000 (<-- my guest output)
# cat /sys/module/hello/sections/.data
0xd0b1a38c (<-- my guest output)
# cat /sys/module/hello/sections/.bss
0xd0b1b580 (<-- my guest output)

After the gdb was connected to the target, We inform it our module offset (the compiled kernel is located at : /home/ofer/hello.ko):
(gdb)add-symbol-file /home/ofer/hello.ko 0xd0b1a000 -s .data 0xd0b1a38c -s .bss 0xd0b1b580

Now I will add breakpoint to the module function:
(gdb)b hello_func
and I will receive the following answer in the gdb output window:
Breakpoint 1 at 0xd0b1a003: file /home/ofer/hello.ko, line 10.

In order to call this function, I will remove the module from the guest console and re-insmod it (which invoke the hello_func function):
# rmmod hello
# insmod hello.ko
The gdb will stop in the required function and we can debug the code.

Prepare and debug linux kernel with vmware 6

Vmware 6 includes kernel-debugging option which can replaced the kdb usage, Here's some explanation of this process:

The kernel source preparation:

Before the debugging we will have to prepare the source code and the debug image of the “debugged” kernel.

In my target machine (Inside the VMware) I'm using linux kernel 2.6.25.9-76.fc9.i686, first we will need to download the source code of this version and the vmlinux image file with debug symbols.
Thedownload location for this release is :
http://kojipkgs.fedoraproject.org/packages/kernel/2.6.25.9/76.fc9/,
From there I downloaded the following files:

• i686/kernel-devel-2.6.25.9-76.fc9.i686.rpm
  

• i686/kernel-debuginfo-common-2.6.25.9-76.fc9.i686.rpm
  

• i686/kernel-debuginfo-2.6.25.9-76.fc9.i686.rpm
  

• src/kernel-2.6.25.9-76.fc9.src.rpm
  

Next, we need to install those packages, but before, We need to create the build directory tree for the source, The next command will create the SOURCES, SPECS and BUILD directories under ${HOME}/rpmbuild
#rpmdev-setuptree
Package installation (root user) :
# rpm -i <the packages>

Now, we need to prepare the source directories with all the patches:

# cd ~/rpmbuild/SPECS
# rpmbuild -bp --target=`uname -m` kernel.spec

The VMware 6 preparation:

Open the VMware configuration file (/etc/vmware/config) and add the following line:

debugStub.listen.guest32=1

Now, We will see the following line in the vmware log file, which mean that we are ready for debug:

VMware Workstation is listening for debug connection on port 8832.

The GDB preparation:

Instead of running the gdb with all the debugging arguments every time, we can edit the .gdbinit file with the initial configuration,
This file should be located either in the current directory or in the
home directory.
Add to the .gdbinit the following line to determine the file to launch:
file /usr/lib/debug/lib/modules/2.6.25.9-76.fc9.i686/vmlinux

Add the .gdbinit the following line in order to tell gdb to connect the remote host:

target remote localhost:8832

Now we will configure the source-code directories, After the previous
steps, the source code is located under ${HOME}/rpmbuild/BUILD/kernel-2.6.25/linux-2.6.25.i686 , to automatically insert all the sub directories, run the following command on bash:

#for file in `find ~/rpmbuild/BUILD/kernel-2.6.25/linux-2.6.25.i686 -type d`; do echo dir $file;done >> ~/.gdbinit

Debugging !!!

Now, all we need to do is to type
#gdb
(or ddd) In order to launch the debugger.

One issue I faced is that if I'm pausing the vmware guest machne to long, its shut down and I need to restart the guest OS again, still need to figure out this one ........