I put it on my blog, so at least some people could have the alternative place instead of digging kernelnewbies archieves. This is a raw summary of discussion between me and Robert P.J. Day. First, an extensive explanation from Robert:
ok, here's what i've learned so far, and i have to admit, a little
of it surprises me. to explain what i'm doing, i've just started to
write a tutorial on kernel debugging for one of my clients, and i'm
trying to start with the absolutely simplest possible techniques.
the simplest method i know of is just
# gdb vmlinux /proc/kcore
AFAIK, unless you have at least the kernel image, there's not much you
can do (but if there is, feel free to fill me in and i'll add it to my
list).
so, as a first attempt, i used the latest git tree and explicitly
configured *without* the DEBUG_INFO selection. even though LDD3 (p.
100) claims that you need that option to have symbol information,
that's not entirely true.
once i configured and built my kernel, i then had my vmlinux file
and my System.map file, and i rebooted under that new kernel. i could
then compare the contents of System.map to the contents of
/proc/kallsyms, just to verify that it looked sane:
$ grep "D jiffies" System.map
c054bc00 D jiffies
c054bc00 D jiffies_64
$ grep "D jiffies" /proc/kallsyms
c054bc00 D jiffies
c054bc00 D jiffies_64
ok, looks good. and, at this point, as root, i can do this:
# gdb vmlinux /proc/kcore
...
warning: shared library handler failed to enable breakpoint
Core was generated by `ro root=/dev/fc5/root rhgb quiet'.
#0 0x00000000 in ?? ()
(gdb) p jiffies
$1 = 1084958
(gdb) p max_cpus
$2 = 32
(gdb)
...
so even eithout selecting DEBUG_INFO, you can still examine at
least *basic* data objects. what DEBUG_INFO gives you (as i read it)
is the ability to dump more complicated objects, like structures. but
even without that feature, gdb is still moderately useful.
i had always read that you absolutely needed DEBUG_INFO to use gdb
in any useful way, but it's clear that that's not true.
----------------------------------------------------------------------
And I say:
> ok, here's what i've learned so far, and i have to admit, a little
> of it surprises me. to explain what i'm doing, i've just started to
> write a tutorial on kernel debugging for one of my clients, and i'm
>
Instead of "debugging" in its true meaning (dumping values, setting breakpoint, observing stack frames, and so on), if you just use gdb that way (without using kgdb, kdb and etc) we can only dump variables, or possibly anything that just related to passive observation.
And one thing (I just tested moment ago), seems like gdb "caches" the result of "print" command. This is probably related to the fact that kcore is dynamically changed but gdb only check the value of the startup stage. So, "jiffies" or any other dynamic variables seems constant.
> trying to start with the absolutely simplest possible techniques.
>
> the simplest method i know of is just
>
> # gdb vmlinux /proc/kcore
>
> AFAIK, unless you have at least the kernel image, there's not much you
> can do (but if there is, feel free to fill me in and i'll add it to my
> list).
>
>
$ grep jiffies /boot/System.map
c0354644 B jiffies
^^^^^
convert that value to decimal, because AFAIK dd can not accept offset in hexadecimal form.
$ dd if=/dev/kmem skip=3224716868 bs=1 count=4 | od -t uL
We fetch 4 bytes, since jiffies (not jiffies64) is an unsigned long variable. We tell od to display it as unsigned long as well.
For more about this kind of technique, google for "kernel memory forensics".
I hope that recipe is correct. feel free to try that....
----------------------------------------------------------------------
Robert eventually reminds me of this:
>seems like gdb "caches" the result of "print" command.
yes, which is why you need to re-load the core file each time with:
(gdb) core-file /proc/kcore
that will then show you the latest value. try it, you'll see.
----------------------------------------------------------------------
I hope that is a useful info for your all.
regards,
Mulyadi
04 July 2007
12 June 2007
Copy of feedback Jeff Dike gave me...
Hello everyone. Recently I wrote about GCC on Onlamp and some folks gave me feedback. I believe this will be a valuable piece for everybody, so I put it on my blog. The same comment is also posted in Onlamp, but I screwed up the HTML output. For those who got trouble reading the text there, I put the corect version here.
This one I got from Jeff. Sharp criticism... The text written in italic is my original text, followed with the comment.
gcc (GNU C Compiler) is actually a collection of frontend tools that
Actually, gcc == GNU Compiler Collection - the whole family is referred to as gcc.
Preprocessing: Producing code that no longer contains directives. Things like "#if" cannot be understood directly by the compiler, so this must be translated into real code. Macros are also expanded at this stage, making the resulting code larger than the original.
It also pulls in headers.
..manipulate them further. This work is done in multipass style, which demonstrates that it sometimes takes more than one scan through the source code to optimize.
It doesn't scan the source - it scans the intermediate format, which used to be RTL, but which is something else now.
...As you may already be aware, registers can be accessed hundreds or thousands times faster than RAM cells.
Exaggeration - Maybe ~100 cycles for going out to main memory, but these things will be in cache, so might cost a few cycles.
0x7530 is 30,000 in decimal form, so we can quickly guess the loop is..
0x7530 is hex, "0x7530 is 30,000 in hexadecimal form" or "0x7530 in decimal is 30,000"
simplified. This code represents the innermost loop and the outermost loop ("for(j=0;j<5000;j++) ... for(k=0;k<4;k++)") because that is literally a request to do 30,000 loops. Note that you just need to...
5000 * 4 = 20000 loops.
Author's note: I admit this is solely my own mistake that confused number of loops with the current value of accumulator (acc variable). The correct sentence should be "this code represents the middle and the innermost loop (for(j=0;j<5000;j++) ... for(k=0;k<4;k++)). In the end of these loops, accumulator is increased by 30,000".
To illustrate them better, here are the codes with inline comments. First check #1, then #2 and so on to understand the flow.
80483a6: jmp 80483c7 <main+0x37>
80483a8: add $0x7530,%ecx 4. acc += 30,000 ?
80483ae: cmp $0x11e1a300,%ecx 5. accumulator has reached 300,000,000 ?
80483b4: je 80483d0 <main+0x40>
80483b6: jmp 80483c7 <main+0x37>
80483b8: add $0x6,%edx 2. the innermost loop.
80483bb: add $0x1,%eax EAX is the counter for middle loop.
80483be: cmp $0x1388,%eax 3. have we loop 5,000 times yet?
80483c3: je 80483a8 <main+0x18>
80483c5: jmp 80483b8 <main+0x28>
80483c7: mov %ecx,%edx 1. starts here.
80483c9: mov $0x0,%eax
80483ce: jmp 80483b8 <main+0x28>
80483d0: mov %edx,0x4(%esp) 6. ready to print.
80483d4: movl $0x80484a0,(%esp)
80483db: call 80482b8 <printf@plt>
So, instead of originally looping 200,000,000 (10,000 * 5,000 * 4) times, it now does 50,000,000 (10,000 * 5,000) times only.
Now, on to parameter passing. In x86 architectures, parameters are pushed to the stack and later popped inside the function for further processing.
Sometimes popped, often they are left on the stack.
By using -mregparm, you basically break the Intel x86-compatible Application Binary Interface (ABI). Therefore, you should mention it when you distribute your software in binary only form.
Why? I see no problem shipping source with Makefiles that say -mregparam. The ABI problem comes if you were to redeclare a library function as regparam and call it.
09 February 2007
The Man named Ulrich Drepper
You don't know who he is ? Just for intro, he is one the Glibc maintainer. So this guy deals with one of the most wanted component inside Linux. But what I found interesting from Mr. Drepper isn't really about his skill, but how he comments on something.
For example, taken from his posting:
"If the title promises the
FYI, the context of the above sentence was when he attended a seminar's session conducted by Eugene Kaspersky ( a familiar name for you? yes, this guy writes anti virus). So, as you can see, Mr. Drepper highly criticized the "mismatch" between the session's title and the actual materials. All I can say, Mr. Drepper is very hard-to-pleased :) And he called this session was a baloney...oh my! :)
How about this?
"There are two ways I can interpret Steve's comments:
More interesting right? Oh before I forgot, you can read them all from Ulrich Drepper journal. Ok back, Mr Drepper "clearly and honestly" told us that this Steve G had no adequate idea on how NX bit really works, what it can prevent and what it can't. And certainly, a return-to-libc is simple enough to defeat this if you found a buffer overflow case.
All in all, I found Mr Drepper as highly technical but also a quite verbose thinker kind of man. Absolutely no holding back when speaking his mind. Some people (including me) do like this style, but the rest are not. Personally, via this blog, I suggest to Mr. Drepper to calm down a bit and find better wording to criticize those morons. OK, now I am really rude :))
yours truly,
Mulyadi Santosa
For example, taken from his posting:
"If the title promises the
latesttactics, why waste time on ancient history? When promising
details, why only scratch the surface and throw out a few buzzwords? This was probably one of the most wasteful hour I've spent in a long time. Heck, I might have enjoyed an HR seminar more than this baloney."
FYI, the context of the above sentence was when he attended a seminar's session conducted by Eugene Kaspersky ( a familiar name for you? yes, this guy writes anti virus). So, as you can see, Mr. Drepper highly criticized the "mismatch" between the session's title and the actual materials. All I can say, Mr. Drepper is very hard-to-pleased :) And he called this session was a baloney...oh my! :)
How about this?
"There are two ways I can interpret Steve's comments:
- On Windows, because it is such a soft target, attackers didn't have to bother with more sophisticated attacks and they really didn't happen. In this situation the attackers will simply adapt and use the attack vectors I described above.
- Steve doesn't know what he's talking about and he's doing his listeners a disservice by suggesting they are almost completely safe just because they enable NX.
More interesting right? Oh before I forgot, you can read them all from Ulrich Drepper journal. Ok back, Mr Drepper "clearly and honestly" told us that this Steve G had no adequate idea on how NX bit really works, what it can prevent and what it can't. And certainly, a return-to-libc is simple enough to defeat this if you found a buffer overflow case.
All in all, I found Mr Drepper as highly technical but also a quite verbose thinker kind of man. Absolutely no holding back when speaking his mind. Some people (including me) do like this style, but the rest are not. Personally, via this blog, I suggest to Mr. Drepper to calm down a bit and find better wording to criticize those morons. OK, now I am really rude :))
yours truly,
Mulyadi Santosa
23 December 2006
An enlightenment about shadow page table
Lately, I got trouble understanding why Virtual Machine Monitor (hypervisors, such as Xen) implement shadow page table. So I entered #osdev in Freenode and got these explanations from these generous gentlemen. Enjoy! (Note: the_hydra is /me)
<the_hydra>hi
<the_hydra> could somebody help me understanding what shadow page
table really is?
<the_hydra> from what I read, seems like we do shadow because CPU in
vmx root mode doesn't care with guest mode PTEs
<the_hydra> while guest only "sees" the guest mode PMD/PGD/PTEs, is
this correct?
* schoolboy has joined osdev
<geist> I assume you're talking about intel's VT stuff?
<the_hydra> geist: yes
<the_hydra> sorry was afk
<the_hydra> vt-i and vt-x if I might add
* KillerX has joined osdev
<the_hydra> geist: care to explain?
<geist> dont know enough about the intel variant
<geist> i know enough to know they screwed it up
<the_hydra> oh :|
<the_hydra> ok maybe you can explain in general how shadow page table
works?
<geist> i dont know enough details to give you a reasonable explanation
<geist> i read the spec on the amd design, but only have heard about
the intel one
<geist> and the intel one is a lot more crappy, from what I hear
<the_hydra> hm ok np
* redblue has quit IRC (Read error: 110 (Connection timed out))
<geist> the amd design completely virtualizes it, so the guest doesn't
have to care about the higher level page tables
<the_hydra> sounds great!
<geist> the intel one doesn't completely hide the physical pages
<geist> so it's very hard to make a perfectly secure system
<the_hydra> so in AMD's, VMM just need initially tell where to store
real and "fake" pgd pointer and the rest will be taken care by CPU?
<geist> that's what i understand, yeah
* schoolboy has quit IRC ("hello world")
* wcstok has quit IRC (Remote closed the connection)
* Mikaku has quit IRC ("Leaving")
* _anoid has joined osdev
<mwk> geist: my support. intel VMX sucks.
<the_hydra> mwk: you think so too?
<mwk> the_hydra: AMD SVM system may or may not support Nested Paging,
according to the specification [i don't have any idea if it's
actually supported in RL processors or not, though]
<mwk> if CPU supports nested paging, you have host CR3 and guest CR3
<the_hydra> mwk: oh so the official name for this hardware based MMU
virtualization is called nested paging?
<mwk> guest CR3 is just the virtualised machine's linear-to-physical
translation, so it can be taken directly from guest's virtualised CR3
<mwk> yeah
<mwk> host CR3 provides guest-physical-to-host-physical translation
<mwk> so, you manage host CR3 and let virtualised guest manage guest
CR3
<the_hydra> hm
<mwk> but, if CPU supports SVM and not nested paging, you need shadow
paging tables
<mwk> which provide guest-linear-to-host-physical translation directly
* wobster has joined osdev
<the_hydra> with this "double" mapping (guest virt to guest phys,
guest phys to host phys) ...do you think it will have impact in
virtualization? perfomance... latency and so on
<mwk> not much
<mwk> but it'll help
<mwk> so, if you need shadow tables, you do the following:
<mwk> 1. create empty page table
<mwk> 2. run the guest
<mwk> 3. make CR3 read/write, invlpg, and page fault interceped events
<the_hydra> sorry what is invplg?
<mwk> invlpg.
<the_hydra> *lpg*
<mwk> leave now.
<the_hydra> sorry what is invlpg??
<mwk> uhm... did you read the manual about paging?
<mwk> anyway:
<mwk> INVLPG Invalidate TLB Entry
<mwk> so
<mwk> 4. when VM exits due to nonexistent-page fault, check CR2 and
walk guest and host page tables to see if it actually has some
translation. if so, insert it to shadow page table and restart VM.
otherwise, inject real page fault into VM
<the_hydra> ok got it...wasn't familiar with that invlpg, but I do
understand what invalidate TLB entry is
<mwk> 5. when VM exits due to invlpg, just zero out that entry in
shadow tables
<mwk> 6. when VM exits due to CR3 write [or CR0 or CR4, in fact],
delete all shadow tables and replace with an empty one
<mwk> that should be it
<the_hydra> very detailed...thanks a lot
<mwk> oh, also, when you intercept invlpg and/or CR3 write, you need
to flush real CPU's TLBs... so you need invlpga, or ASID change.
details are in AMD spec.
21 December 2006
Feel happy when you found your ideas are well appreciated
Dear readers....
I think I read a very good news. I visited del.icio.us and I found this. By the time I read that, my article has been bookmarked by 158 people...and all I can say is "Wow"..
Of course, the fee is good, but looking the fact so many people found my article useful is indeed a greater happiness. And who I should thank to? It's not other than GOD. Honestly, initially this idea sound silly even to me...but something deep inside me convinced me it worth and I should try. So, I try, Mr. Chromatic from O'reilly approved it, I worked on it for about 1 month and it got published.
"Do the best and God will do the rest"...
regards,
Mulyadi
I think I read a very good news. I visited del.icio.us and I found this. By the time I read that, my article has been bookmarked by 158 people...and all I can say is "Wow"..
Of course, the fee is good, but looking the fact so many people found my article useful is indeed a greater happiness. And who I should thank to? It's not other than GOD. Honestly, initially this idea sound silly even to me...but something deep inside me convinced me it worth and I should try. So, I try, Mr. Chromatic from O'reilly approved it, I worked on it for about 1 month and it got published.
"Do the best and God will do the rest"...
regards,
Mulyadi
20 November 2006
Vini Vidi Vici (participating in local Linux Troubleshooting competition)
Thursday, November 16th 2006, another normal day for me. OK, that's wrong :) Like any other IT folks nowadays, I spent some part of my free time to browse the Internet. Guess what I found? What? Attending the conference? No no, that just made me feel sleepy and awful. Think again. Yup, I decided to enter the Linux trouble-shooting competition. First thing that came inside my brain was "what the hell I shall face there? A God damn broken ftp server? Compromised host with some sort of 3133t r00k1ts installed there?Awfully broken root filesystem? Or what?" Yeah, that sounds scary, but anything could be the challenge I shall face, so I must be ready for anything. What will I lose anyway? Nothing... except my 50,000.00 rupiahs will fly away :)) hehehehe.
But shit, the end of the registration was near. F**k! So, I picked up my cell phone and dialled the CP (contact person) number. Pheww, the man on the phone said, anybody was still allowed to register 'til the competition was about to begin (that was November 18th, Saturday...). Yeeha. But oh my... registered as a team? Hm, I couldn't ask somebody to join me, so I simply asked again "ehm, I plan to register as a team composed of only one member, is it OK?". Quite funny part here, since I got confusing answer at first. But eventually I got another positive answer again.. Pheww, so the only thing I need to worry is the competition itself.
Saturday, got up earlier, cleaned up myself, got some food and I went to Surabaya. No traffic jam, thank God for that! Arrived at Hi-Tech Mall (a.k.a THR -- Taman Hiburan Remaja), wait for couple minutes, went to 2nd floor and I registered myself. After me, I saw many people coming and did on site registration. And I thought "...surely lack of publication....", but that was just my guess.
Now the hard part... show time baby! :)
In the schedule, the competition should began at 10 AM, but well, that was the theory :) The fact was, it began close to 11 AM :) Surely pumping up your adrenalin? No, quite the opposite, I began to feel sleepy :)))) bwahahahahha. I used this spare time to look around and try to "profile" the other competitors. Holy cow! I was pretty damn sure I was the oldest competitor :) Gee, time flies so fast isn't it? Once you were 17, and now you're suddenly near 28 and your brain is working slower ... shit on me :D
First round. Me and the other 9 time must solved the boot problem. The sympton was quite simply but could be tricky if you didn't understand the real issue. After an item in GRUB was selected, the kernel was booted (NB: the distro used here was Blank On, a localized and modified version of Fedora Core). After some hardware detection, suddenly it went to run-level 6. Did you notice the quirk here? yeap, certainly something was forcing the init stage to go to runlevel 6. It could be the kernel's boot param, or... /etc/inittab. So, I inspected the related GRUB's entry. Clear, nothing was wrong there. Then how to go to the normal runlevel without interrupted by the inittab? Simply provide your own runlevel number as the kernel boot param, e.g linux 3 root=/dev/hda2 and you're done! But since I must made it permanent, I told the kernel to enter single mode (passing "S" as boot parameter), edit /etc/inittab and made 5 as initial run level. Save the file, reboot and voila, I enter the GDM :)
taking a break for about an hour or so, I entered the 2nd round. In this round, 10 from about total 20 team was selected since they solved the first round and wrote the explanation (problems identification, solution) pretty clearly. Quite good judging style, I did like it. So back to the arena. Now what? Sounds simple, X couldn't be started! All I got was a blank shitty screen when I executed "startx". And thanks to this lame head, I forgot where the hell X saved its log! For once, I thought strace could be the life saver here, but .. great... no strace! So it's kinda back to stone age where the tool was just the rock (got my point?)
My mind quickly concluded it could be some sort of permission problem. So I switch to another user ID. What did I got? A message telling me that I had no permission to start X and big chance it was PAM problem (according to the on screen message). So I tried to follow the hint and inspected the PAM configurations. Coming inside xserver PAM setting, changing here and there, no luck. Bummer! What did I wrong here... ??
Almost losing hope here, while one team was announced as the first team correctly solve the problem. Hm, quite fast, so it must be something easy. Minutes later, the announcer told a quite useful hint : "the problem is something related to mouse device!" OK, X and mouse, what could be the link here? Of course, Xorg.conf (small x? big X? I forgot..)! Before that, thanks to "find", I finally found the X log...and yep, it was said that core pointer couldn't be initialized. What's the problem? Could it be the mouse wasn't detected? dmesg confirmed that the PS/2 mouse was indeed correctly detected, so that wasn't the problem. Firing up vi to inspect Xorg.conf, I looked up any directives that could lead to mouse settings. OK, I found the device name to be used by X as the pointer..."/dev/input/mouse". Valid entry? Not really, ls confirmed that device name was wrong because of missing the "0" suffix. The correct one was /dev/input/mouse0. Typing that, saved, fired up startx again and I was done! :)
Third round (final one) was started at 4 PM...oh man, I told you, I was so tired and sweating... and I thought to myself (not a song :) ) "this is it, exhausted...can I win?". But giving up isn't my style... the only style I have is "keep moving!". So that's what I did. This time, 5 team entered the final... and they were all looked skillful, so I planned to give my best here. And the game was started again, this time...three troubles...all at once! F**k!
First, unable to find certain file when GRUB tried to load the selected kernel. What was it? Quick inspection in grub entries revealed it was incorrect initrd's filename. Fixing that and I met another problem... "no users exist" (something like that, I don't remember 100%). Why? No ideas.... but I reset my PC and back to inspect the GRUB entries. I noticed that it passed /dev/hda5 as the root filesystem. Wrong one? Once again no idea, but I guessed it could be the problem. So I changed it to point to another valid one and I used /dev/hda8 since this rootfs was also used by another kernel's entry. OK, booting went smoothly this time. X was also fixed quite similar like round #2. Confidently I raised my hand to let the observers knew I did it. The judging comittee checked my solution and one of them said "oh, you can't change the root filesystem, please change it back". Great.... :(
At almost the same time, another time raise their hands and the judges declared that they did it correctly. Oh my, I lost my chance as the #1 winner. But I won't give up so easily. No change to rootfs? OK, I can do that too. Rolling back the boot loader's settings to the original one, this time I got the "no user" problem again. I was almost sure the root ID wasn't there, so I used another kernel entry to boot the Linux system and inspect /dev/hda5. Got it, no root! So what I did was simply creating it. And since I am lazy, I did :
# grep root /etc/passwd | head -1 >> /mnt/test/etc/passwd
/dev/hda5 was mounted in /mnt/test and I was in the healthy /dev/hda8. Once again, reboot. This time, using runlevel 3, I successfully entered the login prompt. Login and I tried startx. Mouse problem again? Nope... "no screens available". Sounds like driver problem? I checked the Xorg.conf and found the bug, wrong driver! It was written "Sis" as the display driver while obviosly this machine was using Radeon graphic card (I didn't confirmed it using lspci, I purely used instinct since the other machine was using radeon and it worked). While I hacked it, the other team was declared as the 2nd successful solver. OK, I must secure 3rd place then. Finished with Xorg.conf, starting X and it worked! Again, I raised my hand......
Clock ticking, the judges inspected my PC once again... typing here and there, they were trying to confirm something (correct partition? not sure..). But I was pleased to hear they concluded I made correct solution.
Waiting another 2-3 minutes, the official winners were announced. And yep, I got the 3rd place. I was still proud to myself, knowing that I was still quite fast solving the final problem. And the bonus was nice too, Rp 500,000.00 ... not a small cash to carry, right? :)
So, boys and gals, there you go...the story of a man competing for a pride and honour. And I must say, I do like this kind of competition. Not only because it offers competitive atmosphere and quite nice prize, but also because the participant must *solve* a real world problem. Yes, I said it was real world because I met all those problems previously. Compared to the hacking competition, I found this troubleshooting competition is far more educative, challenging and promoting Linux usage in general. Kudos for the organizer who did the quite good job. I hope there will be another similar competition like this and I shall enter it once again. hopefully facing far more mind-juggling problems to solve.
But shit, the end of the registration was near. F**k! So, I picked up my cell phone and dialled the CP (contact person) number. Pheww, the man on the phone said, anybody was still allowed to register 'til the competition was about to begin (that was November 18th, Saturday...). Yeeha. But oh my... registered as a team? Hm, I couldn't ask somebody to join me, so I simply asked again "ehm, I plan to register as a team composed of only one member, is it OK?". Quite funny part here, since I got confusing answer at first. But eventually I got another positive answer again.. Pheww, so the only thing I need to worry is the competition itself.
Saturday, got up earlier, cleaned up myself, got some food and I went to Surabaya. No traffic jam, thank God for that! Arrived at Hi-Tech Mall (a.k.a THR -- Taman Hiburan Remaja), wait for couple minutes, went to 2nd floor and I registered myself. After me, I saw many people coming and did on site registration. And I thought "...surely lack of publication....", but that was just my guess.
Now the hard part... show time baby! :)
In the schedule, the competition should began at 10 AM, but well, that was the theory :) The fact was, it began close to 11 AM :) Surely pumping up your adrenalin? No, quite the opposite, I began to feel sleepy :)))) bwahahahahha. I used this spare time to look around and try to "profile" the other competitors. Holy cow! I was pretty damn sure I was the oldest competitor :) Gee, time flies so fast isn't it? Once you were 17, and now you're suddenly near 28 and your brain is working slower ... shit on me :D
First round. Me and the other 9 time must solved the boot problem. The sympton was quite simply but could be tricky if you didn't understand the real issue. After an item in GRUB was selected, the kernel was booted (NB: the distro used here was Blank On, a localized and modified version of Fedora Core). After some hardware detection, suddenly it went to run-level 6. Did you notice the quirk here? yeap, certainly something was forcing the init stage to go to runlevel 6. It could be the kernel's boot param, or... /etc/inittab. So, I inspected the related GRUB's entry. Clear, nothing was wrong there. Then how to go to the normal runlevel without interrupted by the inittab? Simply provide your own runlevel number as the kernel boot param, e.g linux 3 root=/dev/hda2 and you're done! But since I must made it permanent, I told the kernel to enter single mode (passing "S" as boot parameter), edit /etc/inittab and made 5 as initial run level. Save the file, reboot and voila, I enter the GDM :)
taking a break for about an hour or so, I entered the 2nd round. In this round, 10 from about total 20 team was selected since they solved the first round and wrote the explanation (problems identification, solution) pretty clearly. Quite good judging style, I did like it. So back to the arena. Now what? Sounds simple, X couldn't be started! All I got was a blank shitty screen when I executed "startx". And thanks to this lame head, I forgot where the hell X saved its log! For once, I thought strace could be the life saver here, but .. great... no strace! So it's kinda back to stone age where the tool was just the rock (got my point?)
My mind quickly concluded it could be some sort of permission problem. So I switch to another user ID. What did I got? A message telling me that I had no permission to start X and big chance it was PAM problem (according to the on screen message). So I tried to follow the hint and inspected the PAM configurations. Coming inside xserver PAM setting, changing here and there, no luck. Bummer! What did I wrong here... ??
Almost losing hope here, while one team was announced as the first team correctly solve the problem. Hm, quite fast, so it must be something easy. Minutes later, the announcer told a quite useful hint : "the problem is something related to mouse device!" OK, X and mouse, what could be the link here? Of course, Xorg.conf (small x? big X? I forgot..)! Before that, thanks to "find", I finally found the X log...and yep, it was said that core pointer couldn't be initialized. What's the problem? Could it be the mouse wasn't detected? dmesg confirmed that the PS/2 mouse was indeed correctly detected, so that wasn't the problem. Firing up vi to inspect Xorg.conf, I looked up any directives that could lead to mouse settings. OK, I found the device name to be used by X as the pointer..."/dev/input/mouse". Valid entry? Not really, ls confirmed that device name was wrong because of missing the "0" suffix. The correct one was /dev/input/mouse0. Typing that, saved, fired up startx again and I was done! :)
Third round (final one) was started at 4 PM...oh man, I told you, I was so tired and sweating... and I thought to myself (not a song :) ) "this is it, exhausted...can I win?". But giving up isn't my style... the only style I have is "keep moving!". So that's what I did. This time, 5 team entered the final... and they were all looked skillful, so I planned to give my best here. And the game was started again, this time...three troubles...all at once! F**k!
First, unable to find certain file when GRUB tried to load the selected kernel. What was it? Quick inspection in grub entries revealed it was incorrect initrd's filename. Fixing that and I met another problem... "no users exist" (something like that, I don't remember 100%). Why? No ideas.... but I reset my PC and back to inspect the GRUB entries. I noticed that it passed /dev/hda5 as the root filesystem. Wrong one? Once again no idea, but I guessed it could be the problem. So I changed it to point to another valid one and I used /dev/hda8 since this rootfs was also used by another kernel's entry. OK, booting went smoothly this time. X was also fixed quite similar like round #2. Confidently I raised my hand to let the observers knew I did it. The judging comittee checked my solution and one of them said "oh, you can't change the root filesystem, please change it back". Great.... :(
At almost the same time, another time raise their hands and the judges declared that they did it correctly. Oh my, I lost my chance as the #1 winner. But I won't give up so easily. No change to rootfs? OK, I can do that too. Rolling back the boot loader's settings to the original one, this time I got the "no user" problem again. I was almost sure the root ID wasn't there, so I used another kernel entry to boot the Linux system and inspect /dev/hda5. Got it, no root! So what I did was simply creating it. And since I am lazy, I did :
# grep root /etc/passwd | head -1 >> /mnt/test/etc/passwd
/dev/hda5 was mounted in /mnt/test and I was in the healthy /dev/hda8. Once again, reboot. This time, using runlevel 3, I successfully entered the login prompt. Login and I tried startx. Mouse problem again? Nope... "no screens available". Sounds like driver problem? I checked the Xorg.conf and found the bug, wrong driver! It was written "Sis" as the display driver while obviosly this machine was using Radeon graphic card (I didn't confirmed it using lspci, I purely used instinct since the other machine was using radeon and it worked). While I hacked it, the other team was declared as the 2nd successful solver. OK, I must secure 3rd place then. Finished with Xorg.conf, starting X and it worked! Again, I raised my hand......
Clock ticking, the judges inspected my PC once again... typing here and there, they were trying to confirm something (correct partition? not sure..). But I was pleased to hear they concluded I made correct solution.
Waiting another 2-3 minutes, the official winners were announced. And yep, I got the 3rd place. I was still proud to myself, knowing that I was still quite fast solving the final problem. And the bonus was nice too, Rp 500,000.00 ... not a small cash to carry, right? :)
So, boys and gals, there you go...the story of a man competing for a pride and honour. And I must say, I do like this kind of competition. Not only because it offers competitive atmosphere and quite nice prize, but also because the participant must *solve* a real world problem. Yes, I said it was real world because I met all those problems previously. Compared to the hacking competition, I found this troubleshooting competition is far more educative, challenging and promoting Linux usage in general. Kudos for the organizer who did the quite good job. I hope there will be another similar competition like this and I shall enter it once again. hopefully facing far more mind-juggling problems to solve.
17 November 2006
Someone is watching you (or your work?) :)
Just a short post...
Wandering in WWW, like usual, and I found this blog entry . So, at least I am happy that someone found my work is indeed useful for him/her :)
In this paper titled "Distributed Software Platforms for Rehabilitating Obsolete Hardware", my Unix Review's article got cited by some folks from Italy.
Hm, this URL is interesting. It's like two ways promotion :) My article indirectly promoted EPCKPT, while EPCKPT was indirectly inspiring my article.
Oh wow, GLUG-chat mailing list once mentioned my article? Gee, missed a lot of fame I was :)
Wanna add more? I think I will keep this posting updated whenever I find new citation, reference or anything toward my articles....
Wandering in WWW, like usual, and I found this blog entry . So, at least I am happy that someone found my work is indeed useful for him/her :)
In this paper titled "Distributed Software Platforms for Rehabilitating Obsolete Hardware", my Unix Review's article got cited by some folks from Italy.
Hm, this URL is interesting. It's like two ways promotion :) My article indirectly promoted EPCKPT, while EPCKPT was indirectly inspiring my article.
Oh wow, GLUG-chat mailing list once mentioned my article? Gee, missed a lot of fame I was :)
Wanna add more? I think I will keep this posting updated whenever I find new citation, reference or anything toward my articles....
Subscribe to:
Posts (Atom)