observing kernel variable's content with gdb

I put it on my blog, so at least some people could have the alternative place instead of digging kernelnewbies archieves. This is a raw summary of discussion between me and Robert P.J. Day. First, an extensive explanation from Robert:

ok, here's what i've learned so far, and i have to admit, a little
of it surprises me. to explain what i'm doing, i've just started to
write a tutorial on kernel debugging for one of my clients, and i'm
trying to start with the absolutely simplest possible techniques.

the simplest method i know of is just

# gdb vmlinux /proc/kcore

AFAIK, unless you have at least the kernel image, there's not much you
can do (but if there is, feel free to fill me in and i'll add it to my
list).

so, as a first attempt, i used the latest git tree and explicitly
configured *without* the DEBUG_INFO selection. even though LDD3 (p.
100) claims that you need that option to have symbol information,
that's not entirely true.

once i configured and built my kernel, i then had my vmlinux file
and my System.map file, and i rebooted under that new kernel. i could
then compare the contents of System.map to the contents of
/proc/kallsyms, just to verify that it looked sane:

$ grep "D jiffies" System.map
c054bc00 D jiffies
c054bc00 D jiffies_64

$ grep "D jiffies" /proc/kallsyms
c054bc00 D jiffies
c054bc00 D jiffies_64

ok, looks good. and, at this point, as root, i can do this:

# gdb vmlinux /proc/kcore
...
warning: shared library handler failed to enable breakpoint
Core was generated by `ro root=/dev/fc5/root rhgb quiet'.
#0 0x00000000 in ?? ()
(gdb) p jiffies
$1 = 1084958
(gdb) p max_cpus
$2 = 32
(gdb)
...

so even eithout selecting DEBUG_INFO, you can still examine at
least *basic* data objects. what DEBUG_INFO gives you (as i read it)
is the ability to dump more complicated objects, like structures. but
even without that feature, gdb is still moderately useful.

i had always read that you absolutely needed DEBUG_INFO to use gdb
in any useful way, but it's clear that that's not true.

----------------------------------------------------------------------

And I say:

> ok, here's what i've learned so far, and i have to admit, a little
> of it surprises me. to explain what i'm doing, i've just started to
> write a tutorial on kernel debugging for one of my clients, and i'm
>
Instead of "debugging" in its true meaning (dumping values, setting breakpoint, observing stack frames, and so on), if you just use gdb that way (without using kgdb, kdb and etc) we can only dump variables, or possibly anything that just related to passive observation.

And one thing (I just tested moment ago), seems like gdb "caches" the result of "print" command. This is probably related to the fact that kcore is dynamically changed but gdb only check the value of the startup stage. So, "jiffies" or any other dynamic variables seems constant.

> trying to start with the absolutely simplest possible techniques.
>
> the simplest method i know of is just
>
> # gdb vmlinux /proc/kcore
>
> AFAIK, unless you have at least the kernel image, there's not much you
> can do (but if there is, feel free to fill me in and i'll add it to my
> list).
>
>
$ grep jiffies /boot/System.map
c0354644 B jiffies
^^^^^
convert that value to decimal, because AFAIK dd can not accept offset in hexadecimal form.

$ dd if=/dev/kmem skip=3224716868 bs=1 count=4 | od -t uL
We fetch 4 bytes, since jiffies (not jiffies64) is an unsigned long variable. We tell od to display it as unsigned long as well.

For more about this kind of technique, google for "kernel memory forensics".

I hope that recipe is correct. feel free to try that....

----------------------------------------------------------------------

Robert eventually reminds me of this:

>seems like gdb "caches" the result of "print" command.

yes, which is why you need to re-load the core file each time with:

(gdb) core-file /proc/kcore

that will then show you the latest value. try it, you'll see.

----------------------------------------------------------------------

I hope that is a useful info for your all.

regards,

Mulyadi