23 December 2006

An enlightenment about shadow page table

Lately, I got trouble understanding why Virtual Machine Monitor (hypervisors, such as Xen) implement shadow page table. So I entered #osdev in Freenode and got these explanations from these generous gentlemen. Enjoy! (Note: the_hydra is /me)

<the_hydra> could somebody help me understanding what shadow page
table really is?
<the_hydra> from what I read, seems like we do shadow because CPU in
vmx root mode doesn't care with guest mode PTEs
<the_hydra> while guest only "sees" the guest mode PMD/PGD/PTEs, is
this correct?
* schoolboy has joined osdev
<geist> I assume you're talking about intel's VT stuff?
<the_hydra> geist: yes
<the_hydra> sorry was afk
<the_hydra> vt-i and vt-x if I might add
* KillerX has joined osdev
<the_hydra> geist: care to explain?
<geist> dont know enough about the intel variant
<geist> i know enough to know they screwed it up
<the_hydra> oh :|
<the_hydra> ok maybe you can explain in general how shadow page table
<geist> i dont know enough details to give you a reasonable explanation
<geist> i read the spec on the amd design, but only have heard about
the intel one
<geist> and the intel one is a lot more crappy, from what I hear
<the_hydra> hm ok np
* redblue has quit IRC (Read error: 110 (Connection timed out))
<geist> the amd design completely virtualizes it, so the guest doesn't
have to care about the higher level page tables
<the_hydra> sounds great!
<geist> the intel one doesn't completely hide the physical pages
<geist> so it's very hard to make a perfectly secure system
<the_hydra> so in AMD's, VMM just need initially tell where to store
real and "fake" pgd pointer and the rest will be taken care by CPU?
<geist> that's what i understand, yeah
* schoolboy has quit IRC ("hello world")
* wcstok has quit IRC (Remote closed the connection)
* Mikaku has quit IRC ("Leaving")
* _anoid has joined osdev
<mwk> geist: my support. intel VMX sucks.
<the_hydra> mwk: you think so too?
<mwk> the_hydra: AMD SVM system may or may not support Nested Paging,
according to the specification [i don't have any idea if it's
actually supported in RL processors or not, though]
<mwk> if CPU supports nested paging, you have host CR3 and guest CR3
<the_hydra> mwk: oh so the official name for this hardware based MMU
virtualization is called nested paging?
<mwk> guest CR3 is just the virtualised machine's linear-to-physical
translation, so it can be taken directly from guest's virtualised CR3
<mwk> yeah
<mwk> host CR3 provides guest-physical-to-host-physical translation
<mwk> so, you manage host CR3 and let virtualised guest manage guest
<the_hydra> hm
<mwk> but, if CPU supports SVM and not nested paging, you need shadow
paging tables
<mwk> which provide guest-linear-to-host-physical translation directly
* wobster has joined osdev
<the_hydra> with this "double" mapping (guest virt to guest phys,
guest phys to host phys) you think it will have impact in
virtualization? perfomance... latency and so on
<mwk> not much
<mwk> but it'll help
<mwk> so, if you need shadow tables, you do the following:
<mwk> 1. create empty page table
<mwk> 2. run the guest
<mwk> 3. make CR3 read/write, invlpg, and page fault interceped events
<the_hydra> sorry what is invplg?
<mwk> invlpg.
<the_hydra> *lpg*
<mwk> leave now.
<the_hydra> sorry what is invlpg??
<mwk> uhm... did you read the manual about paging?
<mwk> anyway:
<mwk> INVLPG Invalidate TLB Entry
<mwk> so
<mwk> 4. when VM exits due to nonexistent-page fault, check CR2 and
walk guest and host page tables to see if it actually has some
translation. if so, insert it to shadow page table and restart VM.
otherwise, inject real page fault into VM
<the_hydra> ok got it...wasn't familiar with that invlpg, but I do
understand what invalidate TLB entry is
<mwk> 5. when VM exits due to invlpg, just zero out that entry in
shadow tables
<mwk> 6. when VM exits due to CR3 write [or CR0 or CR4, in fact],
delete all shadow tables and replace with an empty one
<mwk> that should be it
<the_hydra> very detailed...thanks a lot
<mwk> oh, also, when you intercept invlpg and/or CR3 write, you need
to flush real CPU's TLBs... so you need invlpga, or ASID change.
details are in AMD spec.

21 December 2006

Feel happy when you found your ideas are well appreciated

Dear readers....

I think I read a very good news. I visited and I found this. By the time I read that, my article has been bookmarked by 158 people...and all I can say is "Wow"..

Of course, the fee is good, but looking the fact so many people found my article useful is indeed a greater happiness. And who I should thank to? It's not other than GOD. Honestly, initially this idea sound silly even to me...but something deep inside me convinced me it worth and I should try. So, I try, Mr. Chromatic from O'reilly approved it, I worked on it for about 1 month and it got published.

"Do the best and God will do the rest"...



20 November 2006

Vini Vidi Vici (participating in local Linux Troubleshooting competition)

Thursday, November 16th 2006, another normal day for me. OK, that's wrong :) Like any other IT folks nowadays, I spent some part of my free time to browse the Internet. Guess what I found? What? Attending the conference? No no, that just made me feel sleepy and awful. Think again. Yup, I decided to enter the Linux trouble-shooting competition. First thing that came inside my brain was "what the hell I shall face there? A God damn broken ftp server? Compromised host with some sort of 3133t r00k1ts installed there?Awfully broken root filesystem? Or what?" Yeah, that sounds scary, but anything could be the challenge I shall face, so I must be ready for anything. What will I lose anyway? Nothing... except my 50,000.00 rupiahs will fly away :)) hehehehe.

But shit, the end of the registration was near. F**k! So, I picked up my cell phone and dialled the CP (contact person) number. Pheww, the man on the phone said, anybody was still allowed to register 'til the competition was about to begin (that was November 18th, Saturday...). Yeeha. But oh my... registered as a team? Hm, I couldn't ask somebody to join me, so I simply asked again "ehm, I plan to register as a team composed of only one member, is it OK?". Quite funny part here, since I got confusing answer at first. But eventually I got another positive answer again.. Pheww, so the only thing I need to worry is the competition itself.

Saturday, got up earlier, cleaned up myself, got some food and I went to Surabaya. No traffic jam, thank God for that! Arrived at Hi-Tech Mall (a.k.a THR -- Taman Hiburan Remaja), wait for couple minutes, went to 2nd floor and I registered myself. After me, I saw many people coming and did on site registration. And I thought "...surely lack of publication....", but that was just my guess.

Now the hard part... show time baby! :)

In the schedule, the competition should began at 10 AM, but well, that was the theory :) The fact was, it began close to 11 AM :) Surely pumping up your adrenalin? No, quite the opposite, I began to feel sleepy :)))) bwahahahahha. I used this spare time to look around and try to "profile" the other competitors. Holy cow! I was pretty damn sure I was the oldest competitor :) Gee, time flies so fast isn't it? Once you were 17, and now you're suddenly near 28 and your brain is working slower ... shit on me :D

First round. Me and the other 9 time must solved the boot problem. The sympton was quite simply but could be tricky if you didn't understand the real issue. After an item in GRUB was selected, the kernel was booted (NB: the distro used here was Blank On, a localized and modified version of Fedora Core). After some hardware detection, suddenly it went to run-level 6. Did you notice the quirk here? yeap, certainly something was forcing the init stage to go to runlevel 6. It could be the kernel's boot param, or... /etc/inittab. So, I inspected the related GRUB's entry. Clear, nothing was wrong there. Then how to go to the normal runlevel without interrupted by the inittab? Simply provide your own runlevel number as the kernel boot param, e.g linux 3 root=/dev/hda2 and you're done! But since I must made it permanent, I told the kernel to enter single mode (passing "S" as boot parameter), edit /etc/inittab and made 5 as initial run level. Save the file, reboot and voila, I enter the GDM :)

taking a break for about an hour or so, I entered the 2nd round. In this round, 10 from about total 20 team was selected since they solved the first round and wrote the explanation (problems identification, solution) pretty clearly. Quite good judging style, I did like it. So back to the arena. Now what? Sounds simple, X couldn't be started! All I got was a blank shitty screen when I executed "startx". And thanks to this lame head, I forgot where the hell X saved its log! For once, I thought strace could be the life saver here, but .. great... no strace! So it's kinda back to stone age where the tool was just the rock (got my point?)

My mind quickly concluded it could be some sort of permission problem. So I switch to another user ID. What did I got? A message telling me that I had no permission to start X and big chance it was PAM problem (according to the on screen message). So I tried to follow the hint and inspected the PAM configurations. Coming inside xserver PAM setting, changing here and there, no luck. Bummer! What did I wrong here... ??

Almost losing hope here, while one team was announced as the first team correctly solve the problem. Hm, quite fast, so it must be something easy. Minutes later, the announcer told a quite useful hint : "the problem is something related to mouse device!" OK, X and mouse, what could be the link here? Of course, Xorg.conf (small x? big X? I forgot..)! Before that, thanks to "find", I finally found the X log...and yep, it was said that core pointer couldn't be initialized. What's the problem? Could it be the mouse wasn't detected? dmesg confirmed that the PS/2 mouse was indeed correctly detected, so that wasn't the problem. Firing up vi to inspect Xorg.conf, I looked up any directives that could lead to mouse settings. OK, I found the device name to be used by X as the pointer..."/dev/input/mouse". Valid entry? Not really, ls confirmed that device name was wrong because of missing the "0" suffix. The correct one was /dev/input/mouse0. Typing that, saved, fired up startx again and I was done! :)

Third round (final one) was started at 4 PM...oh man, I told you, I was so tired and sweating... and I thought to myself (not a song :) ) "this is it, exhausted...can I win?". But giving up isn't my style... the only style I have is "keep moving!". So that's what I did. This time, 5 team entered the final... and they were all looked skillful, so I planned to give my best here. And the game was started again, this time...three troubles...all at once! F**k!

First, unable to find certain file when GRUB tried to load the selected kernel. What was it? Quick inspection in grub entries revealed it was incorrect initrd's filename. Fixing that and I met another problem... "no users exist" (something like that, I don't remember 100%). Why? No ideas.... but I reset my PC and back to inspect the GRUB entries. I noticed that it passed /dev/hda5 as the root filesystem. Wrong one? Once again no idea, but I guessed it could be the problem. So I changed it to point to another valid one and I used /dev/hda8 since this rootfs was also used by another kernel's entry. OK, booting went smoothly this time. X was also fixed quite similar like round #2. Confidently I raised my hand to let the observers knew I did it. The judging comittee checked my solution and one of them said "oh, you can't change the root filesystem, please change it back". Great.... :(

At almost the same time, another time raise their hands and the judges declared that they did it correctly. Oh my, I lost my chance as the #1 winner. But I won't give up so easily. No change to rootfs? OK, I can do that too. Rolling back the boot loader's settings to the original one, this time I got the "no user" problem again. I was almost sure the root ID wasn't there, so I used another kernel entry to boot the Linux system and inspect /dev/hda5. Got it, no root! So what I did was simply creating it. And since I am lazy, I did :
# grep root /etc/passwd | head -1 >> /mnt/test/etc/passwd
/dev/hda5 was mounted in /mnt/test and I was in the healthy /dev/hda8. Once again, reboot. This time, using runlevel 3, I successfully entered the login prompt. Login and I tried startx. Mouse problem again? Nope... "no screens available". Sounds like driver problem? I checked the Xorg.conf and found the bug, wrong driver! It was written "Sis" as the display driver while obviosly this machine was using Radeon graphic card (I didn't confirmed it using lspci, I purely used instinct since the other machine was using radeon and it worked). While I hacked it, the other team was declared as the 2nd successful solver. OK, I must secure 3rd place then. Finished with Xorg.conf, starting X and it worked! Again, I raised my hand......

Clock ticking, the judges inspected my PC once again... typing here and there, they were trying to confirm something (correct partition? not sure..). But I was pleased to hear they concluded I made correct solution.

Waiting another 2-3 minutes, the official winners were announced. And yep, I got the 3rd place. I was still proud to myself, knowing that I was still quite fast solving the final problem. And the bonus was nice too, Rp 500,000.00 ... not a small cash to carry, right? :)

So, boys and gals, there you go...the story of a man competing for a pride and honour. And I must say, I do like this kind of competition. Not only because it offers competitive atmosphere and quite nice prize, but also because the participant must *solve* a real world problem. Yes, I said it was real world because I met all those problems previously. Compared to the hacking competition, I found this troubleshooting competition is far more educative, challenging and promoting Linux usage in general. Kudos for the organizer who did the quite good job. I hope there will be another similar competition like this and I shall enter it once again. hopefully facing far more mind-juggling problems to solve.

17 November 2006

Someone is watching you (or your work?) :)

Just a short post...

Wandering in WWW, like usual, and I found this blog entry . So, at least I am happy that someone found my work is indeed useful for him/her :)

In this paper titled "Distributed Software Platforms for Rehabilitating Obsolete Hardware", my Unix Review's article got cited by some folks from Italy.

Hm, this URL is interesting. It's like two ways promotion :) My article indirectly promoted EPCKPT, while EPCKPT was indirectly inspiring my article.

Oh wow, GLUG-chat mailing list once mentioned my article? Gee, missed a lot of fame I was :)

Wanna add more? I think I will keep this posting updated whenever I find new citation, reference or anything toward my articles....

30 April 2006

"What does this man do on 1:17 AM"?

Interesting subject isn't it? Ever think something like that when you work late and you found yourself asking this kind of question? Is it somekind of joke? Irony? Self consciousness over something that hog your life? Everybody has an answer, I believe that and you too gotta believe that you can answer it.

So, let's assume someone (or probably me) asks something like that to me... I have many different answers that I can freely pick:

1. I just finished watching "Fight Club" in local TV. I found this movie rather interesting at the beginning, pretty amusing in the middle and so confusing in the end. A man punch himself, even shoot himself? He finally realized that he suffers somekind of chronic double personality problem. Holy shit.... And let me ask you something, Edward Norton...has twin brother played by Brad Pitt, what do you think? This isn't double personality anymore, I call this almost perfect split personality.

But anyway, I got the message, that is "...sometimes you just have to let it go AS IS...". This simple rule is somehow abandoned by modern human. We're just so fucked up by fancy clothes, expensive furniture, elite cars and other stuffs, but we ignore our basic needs. Deep beneath us, you are screaming...looking for this "basic needs", is the fun part, you can't. Why? Why? Simple, because you "just don't let it go AS IS". "Shit, tell me the reason right now or I'll blow your mouth!". Don't waste your time, that won't work. Just ask yourself why....

2. Work here and there. Yeah, I got great news for your my dear readers. My article has been published on Dr Dobbs Journal. "Congratulations man", thank you. I also made myself proud to lurk into Linux Forums early this April. "shit man, you never stop, do you?" No, I am not :) Who the hell you think so you can give me orders?

Now, I begin to prepare another two articles, one will be co-authored with a PhD student from China. Now this what I call "international business", you type something, someone from the other side of the globe read it and add something, over and over....and wallaaaa, an article is ready :) No, this isn't a magic. It is called "creative work", a work that truly depends on how good your brain do the work. Your brain is in good shape? Then you will produce an excellent idea. In bad shape? Then you will produce crap. Always creates crap? Then you're doomed. Pack your stuffs and move to the cemetary... there, you don't need to use brain, you just need to confess your sin and hope you don't get burned in hell :)

3. I want to talk with The Devil. No, not with that lowly devil, I shoot for the big one. Yeah, who else, with the great Lucifer itself. In fact, I do have an imaginary chat session with Lucifer

Lucifer: Knock knock, big boy ;)
me: Oh damn, here you are. Welcome...
Lucifer: Still serving Him?
me: Do I have other choice?
Lucifer: Shit man, you have OTHER choice...just follow me...
me: sounds good, what do you offer, anyway?
Lucifer: You ask me what I will offer? Don't you know that I offer this whole world to Jesus when I met Him on the desert?
me: Ah yes, you offer Him this world. Same thing you want to offer now?
Lucifer: Of course, what else do you want?
me: No, thanks. This world is just full with lies. I can't handle a bag of lies, and want to hand me a "globeful" of lies? That ruins my day...
Lucifer: Smart answer, johny boy. OK, I must go now, catch you later...
me: yeah, don't forget to close the door, will you..?
Lucifer: Which door? Oh yeah, that "door", the door between human and its dark side. I'll just leave it open, so I can sneak it anytime..
me: whatever....

4. Thinking about girls. Don't think dirty, ok? I think it is normal, male think about sex in every hour of his life, so it's like a second nature :) An ideal girl is like combination of Jessica Alba + Shakira + Beyonce Knowles + Zhang Zi Yi, put them together inside a body and you won't blink even slight second looking this female :)) Woohoo ....

5. I think I need to plan for another cinema visit. "Ultraviolet" is now playing in cinemas and it is directed by Kurt Wimmer, same guy who directed "Equilibrium", one of my all time favourite movie. I read that Kurt invents another style of fight. That must be good, because he did the same thing and invents Gun Kata. Another interesting thing is, Mila Jovovich is the leading actress. I don't know why Kurt pick her but I heard Kurt wrote the script to suit Mila's character. Either Mila is so charming or Mila is a potential actress, I dunno.. Let's find out once I watch it.

So, that's it my friend, 5 possible answers I might give you when you ask what I do on 1:17 AM. Maybe I give you one answer, two, three or five, nobody knows. It is called randomness, enthrophy, chaos. You can't predict my mine, I can't predict yours, so we are both in a complex random situation...

I must end this before you get heart attack...ciao.....