My article goes Arabic!

Without further comment, here it is http://arlinux.110mb.com/lgazet/vim-m.php. It's actually an Arabic translation of LG 2 cent tip written by me about couple months ago in 2007.

And Happy Idul Adha...

regards,

Mulyadi

Finding frame rate tester for Linux?

Breaking from my usual routines on posting Linux internals issues, I post something different right now. Since I am also a (near hardcore) gamer, I always pay attention on recent game releases and development. Well, not so intense like the old days, but I still keep my eyes on it.

And for gamers like me, what's the important thing to watch for? Cool GPU? Of course! Feature rich multimedia library? That's another good score. Free but excellent OS as gaming platform? Of course (actually this is a long way to say Linux has bright future in gaming industry). But, in the end, you need a method to determine the perfomance of those pieces combined into a PC. And that's my friend...is our beloved frame rate tester.

In Windows world, finding such tool isn't too hard. FRAPS is a nice example and I bet is the current standart tool for doing GPU benchmark on Windows. It can do fps (frame per second) calculation (min, max, average), do single frame capture, do continous frame captures and save them as movie (raw AVI if I read the FAQ correctly. Eventually you need to convert it as mpeg or divx one to avoid bloating your precious disk space).

So, how about Linux? Wandering in WWW gave me an answer. Looks like the folks at Anandtech had done the job. Framegetter is a BSD licensed to do more or less what FRAPS does. I can't really judge this tool, but from its short intro article, I think it's enough as basic GPU benchmarking tool. The only thing that made me a bit hesitate to try it is the information that the tool will overwrite(?) the games executables. Hmmm, not so polite IMHO. But maybe, what they mean is probably doing something like external function redirection through LD_PRELOAD trick. In that's case, the information is surely misleading. But all in all, I am happy to see this kind of Linux based tool arise in the surface.

Last note: if you wonder how FRAPS work on Windows, maybe you can check Taksi, an open source tool for frame tester. Take a look on its source code....i am sure it will be a fascinating journey.

regards,

Mulyadi.

Peeking into kernel internals with SystemTap

Just to remind myself and others who are interested in kernel instrumentation. Long time ago, perhaps we did it via manual code modification followed by kernel recompilation, or...uhm, have you ever done syscall hijacking or something like that (ok, it sounds dirty...) ?

Now, here comes SystemTap. Some people call it Dtrace-clone. You can find it here. And here you can read a simple example on how to trace boot process. Hungry for more something alike? Go to a section of Daniel P. Berrange website dedicated to boot instrumnentation.

regards,

Mulyadi

observing kernel variable's content with gdb

I put it on my blog, so at least some people could have the alternative place instead of digging kernelnewbies archieves. This is a raw summary of discussion between me and Robert P.J. Day. First, an extensive explanation from Robert:

ok, here's what i've learned so far, and i have to admit, a little
of it surprises me. to explain what i'm doing, i've just started to
write a tutorial on kernel debugging for one of my clients, and i'm
trying to start with the absolutely simplest possible techniques.

the simplest method i know of is just

# gdb vmlinux /proc/kcore

AFAIK, unless you have at least the kernel image, there's not much you
can do (but if there is, feel free to fill me in and i'll add it to my
list).

so, as a first attempt, i used the latest git tree and explicitly
configured *without* the DEBUG_INFO selection. even though LDD3 (p.
100) claims that you need that option to have symbol information,
that's not entirely true.

once i configured and built my kernel, i then had my vmlinux file
and my System.map file, and i rebooted under that new kernel. i could
then compare the contents of System.map to the contents of
/proc/kallsyms, just to verify that it looked sane:

$ grep "D jiffies" System.map
c054bc00 D jiffies
c054bc00 D jiffies_64

$ grep "D jiffies" /proc/kallsyms
c054bc00 D jiffies
c054bc00 D jiffies_64

ok, looks good. and, at this point, as root, i can do this:

# gdb vmlinux /proc/kcore
...
warning: shared library handler failed to enable breakpoint
Core was generated by `ro root=/dev/fc5/root rhgb quiet'.
#0 0x00000000 in ?? ()
(gdb) p jiffies
$1 = 1084958
(gdb) p max_cpus
$2 = 32
(gdb)
...

so even eithout selecting DEBUG_INFO, you can still examine at
least *basic* data objects. what DEBUG_INFO gives you (as i read it)
is the ability to dump more complicated objects, like structures. but
even without that feature, gdb is still moderately useful.

i had always read that you absolutely needed DEBUG_INFO to use gdb
in any useful way, but it's clear that that's not true.

----------------------------------------------------------------------

And I say:

> ok, here's what i've learned so far, and i have to admit, a little
> of it surprises me. to explain what i'm doing, i've just started to
> write a tutorial on kernel debugging for one of my clients, and i'm
>
Instead of "debugging" in its true meaning (dumping values, setting breakpoint, observing stack frames, and so on), if you just use gdb that way (without using kgdb, kdb and etc) we can only dump variables, or possibly anything that just related to passive observation.

And one thing (I just tested moment ago), seems like gdb "caches" the result of "print" command. This is probably related to the fact that kcore is dynamically changed but gdb only check the value of the startup stage. So, "jiffies" or any other dynamic variables seems constant.

> trying to start with the absolutely simplest possible techniques.
>
> the simplest method i know of is just
>
> # gdb vmlinux /proc/kcore
>
> AFAIK, unless you have at least the kernel image, there's not much you
> can do (but if there is, feel free to fill me in and i'll add it to my
> list).
>
>
$ grep jiffies /boot/System.map
c0354644 B jiffies
^^^^^
convert that value to decimal, because AFAIK dd can not accept offset in hexadecimal form.

$ dd if=/dev/kmem skip=3224716868 bs=1 count=4 | od -t uL
We fetch 4 bytes, since jiffies (not jiffies64) is an unsigned long variable. We tell od to display it as unsigned long as well.

For more about this kind of technique, google for "kernel memory forensics".

I hope that recipe is correct. feel free to try that....

----------------------------------------------------------------------

Robert eventually reminds me of this:

>seems like gdb "caches" the result of "print" command.

yes, which is why you need to re-load the core file each time with:

(gdb) core-file /proc/kcore

that will then show you the latest value. try it, you'll see.

----------------------------------------------------------------------

I hope that is a useful info for your all.

regards,

Mulyadi

Copy of feedback Jeff Dike gave me...

Hello everyone. Recently I wrote about GCC on Onlamp and some folks gave me feedback. I believe this will be a valuable piece for everybody, so I put it on my blog. The same comment is also posted in Onlamp, but I screwed up the HTML output. For those who got trouble reading the text there, I put the corect version here.

This one I got from Jeff. Sharp criticism... The text written in italic is my original text, followed with the comment.

gcc (GNU C Compiler) is actually a collection of frontend tools that

Actually, gcc == GNU Compiler Collection - the whole family is referred to as gcc.

Preprocessing: Producing code that no longer contains directives. Things like "#if" cannot be understood directly by the compiler, so this must be translated into real code. Macros are also expanded at this stage, making the resulting code larger than the original.

It also pulls in headers.

..manipulate them further. This work is done in multipass style, which demonstrates that it sometimes takes more than one scan through the source code to optimize.

It doesn't scan the source - it scans the intermediate format, which used to be RTL, but which is something else now.

...As you may already be aware, registers can be accessed hundreds or thousands times faster than RAM cells.

Exaggeration - Maybe ~100 cycles for going out to main memory, but these things will be in cache, so might cost a few cycles.

0x7530 is 30,000 in decimal form, so we can quickly guess the loop is..

0x7530 is hex, "0x7530 is 30,000 in hexadecimal form" or "0x7530 in decimal is 30,000"

simplified. This code represents the innermost loop and the outermost loop ("for(j=0;j<5000;j++) ... for(k=0;k<4;k++)") because that is literally a request to do 30,000 loops. Note that you just need to...

5000 * 4 = 20000 loops.

Author's note: I admit this is solely my own mistake that confused number of loops with the current value of accumulator (acc variable). The correct sentence should be "this code represents the middle and the innermost loop (for(j=0;j<5000;j++) ... for(k=0;k<4;k++)). In the end of these loops, accumulator is increased by 30,000".

To illustrate them better, here are the codes with inline comments. First check #1, then #2 and so on to understand the flow.

80483a6:        jmp    80483c7 <main+0x37>
80483a8:        add    $0x7530,%ecx 4. acc += 30,000 ?
80483ae:        cmp    $0x11e1a300,%ecx 5. accumulator has reached 300,000,000 ?
80483b4:        je     80483d0 <main+0x40>
80483b6:        jmp    80483c7 <main+0x37>
80483b8:        add    $0x6,%edx 2. the innermost loop.
80483bb:        add    $0x1,%eax EAX is the counter for middle loop.
80483be:        cmp    $0x1388,%eax 3. have we loop 5,000 times yet?
80483c3:        je     80483a8 <main+0x18>
80483c5:        jmp    80483b8 <main+0x28>
80483c7:        mov    %ecx,%edx 1. starts here.
80483c9:        mov    $0x0,%eax
80483ce:        jmp    80483b8 <main+0x28>
80483d0:        mov    %edx,0x4(%esp) 6. ready to print.
80483d4:        movl   $0x80484a0,(%esp)
80483db:        call   80482b8 <printf@plt>

So, instead of originally looping 200,000,000 (10,000 * 5,000 * 4) times, it now does 50,000,000 (10,000 * 5,000) times only.

Now, on to parameter passing. In x86 architectures, parameters are pushed to the stack and later popped inside the function for further processing.

Sometimes popped, often they are left on the stack.

By using -mregparm, you basically break the Intel x86-compatible Application Binary Interface (ABI). Therefore, you should mention it when you distribute your software in binary only form.

Why? I see no problem shipping source with Makefiles that say -mregparam. The ABI problem comes if you were to redeclare a library function as regparam and call it.

The Man named Ulrich Drepper

You don't know who he is ? Just for intro, he is one the Glibc maintainer. So this guy deals with one of the most wanted component inside Linux. But what I found interesting from Mr. Drepper isn't really about his skill, but how he comments on something.

For example, taken from his posting:
"If the title promises the latest tactics, why waste time on ancient history? When promising details, why only scratch the surface and throw out a few buzzwords? This was probably one of the most wasteful hour I've spent in a long time. Heck, I might have enjoyed an HR seminar more than this baloney."

FYI, the context of the above sentence was when he attended a seminar's session conducted by Eugene Kaspersky ( a familiar name for you? yes, this guy writes anti virus). So, as you can see, Mr. Drepper highly criticized the "mismatch" between the session's title and the actual materials. All I can say, Mr. Drepper is very hard-to-pleased :) And he called this session was a baloney...oh my! :)

How about this?
"There are two ways I can interpret Steve's comments:

1. On Windows, because it is such a soft target, attackers didn't have to bother with more sophisticated attacks and they really didn't happen. In this situation the attackers will simply adapt and use the attack vectors I described above.



2. Steve doesn't know what he's talking about and he's doing his listeners a disservice by suggesting they are almost completely safe just because they enable NX.

"

More interesting right? Oh before I forgot, you can read them all from Ulrich Drepper journal. Ok back, Mr Drepper "clearly and honestly" told us that this Steve G had no adequate idea on how NX bit really works, what it can prevent and what it can't. And certainly, a return-to-libc is simple enough to defeat this if you found a buffer overflow case.

All in all, I found Mr Drepper as highly technical but also a quite verbose thinker kind of man. Absolutely no holding back when speaking his mind. Some people (including me) do like this style, but the rest are not. Personally, via this blog, I suggest to Mr. Drepper to calm down a bit and find better wording to criticize those morons. OK, now I am really rude :))

yours truly,

Mulyadi Santosa