How to reduce CPU soft lock up in KVM guest

Ever saw something like below messages inside your KVM (Kernel Virtual Machine) guest's console?

" BUG: soft lockup - CPU#0 stuck for 10s! [swapper:0] "

I did and I find it a bit annoying. If you're inside graphical desktop like environment like KDE or GNOME, you  probably won't notice it directly. But you will likely suffer the same condition, the guest OS somehow become unresponsive for a few moment. In my case, it manifest into stalled CD/DVD access and "ruins" the console display. I had to press Enter few times before I could get back to normal shell prompt. Before I go further, FYI I use Fedora 9, kernel version 2.6.27.23-xx.x.xx.fc9.i686 on a Core Duo powered laptop.

First, why the kernel shows such message? I use the default CentOS 5.3 kernel, so I check the related kernel config inside /boot directory and here is the related configuration item:

CONFIG_DETECT_SOFTLOCKUP=y

What does it do? Ingo Molnar, the writer of this lockup detection patch describe it as:

"From: Ingo Molnar

This patch adds a new kernel debug feature: CONFIG_DETECT_SOFTLOCKUP.

When enabled then per-CPU watchdog threads are started, which try to run once per second. If they get delayed for more than 10 seconds then a callback from the timer interrupt detects this condition and prints out a warning message and a stack dump (once per lockup incident). The feature is otherwise non-intrusive, it doesnt try to unlock the box in any way, it only gets the debug info out, automatically, and on all CPUs affected by the lockup.

Tested this on x86, both with the feature enabled (in which case a provoked lockup was correctly detected) and with the feature disabled. It is CPU-hotplug aware. Should work on every architecture. "

Pay attention here that the watchdog mentioned in the above description has nothing to do with NMI (Non Maskable Interrupt) watchdog. NMI watchdog deal with hard CPU lockup, while the above mentioned lockup watchdog can't. It's just a kernel thread that will stuck if CPU hangs.

I suspect it might be a bug in KVM driver (or specificly, KVM for Intel VT in my case). I came to this hypothesis because the help section of soft lockup patch says:

"Say Y here to enable the kernel to detect "soft lockups", which are bugs that cause the kernel to loop in kernel mode for more than 10 seconds, without giving other tasks a chance to run."

Great...so IMO KVM is too busy on something, or.... something is delaying KVM guest to run. I almost rush to compile my own kernel image using full preemption, hoping that it could squash the problem. But I was tempted to Google a bit more. Interesting result, a post in a mailing list (I forgot which one) suggest to set the CPU frequency into static. Let's try, I edited /etc/sysconfig/cpuspeed so the related lines become:

MAX_SPEED=1333000
MIN_SPEED=1333000

I pick that frequency because it's the middle number between the three available frequencies: 1833000, 1333000 and 1000000 Hz. So theoritically I still get adequate computing power to most job without draining the battery too soon.

Execute:

# service cpuspeed restart

Make sure it's applied correctly:

# grep '1333000' -r /sys/devices/system/cpu/ 
/sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq:1333000
/sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq:1333000

...
/sys/devices/system/cpu/cpu1/cpufreq/scaling_min_freq:1333000
/sys/devices/system/cpu/cpu1/cpufreq/scaling_max_freq:1333000

...

Then I ran my KVM guest again. I did few tasks in it, let it went idle, repeat and so on. During my test for about an hour, the result was promising! It became stable. Well, few lock up happened though, but it was far reduced. Thing that I notice is lockup also happens when I switch into another virtual desktop or if fairly heavy swapping in/out is on the way. So, to further reduce lock-ups, I avoid switching to another virtual desktop (ok, that sucks, but I could live with that) and close any unneccesary applications to conserve virtual memory as much as I can.

Why it works? All I can say is by making the frequency static, you also avoid timer interrupt delivery frequency being changed too. It stabilize the kernel timing and also indirectly stabilize the KVM guest timing. Previously, I was using ondemand power governor and as you might be aware of, it adapts the CPU frequency according to the load quite aggresively. So, frequency was juggling between all three available frequencies. Conservative governor didn't lend a help here. Pity, I assume KVM still force conservative governor to switch to highest frequency then drop most of the time because most of the codes could be run natively instead of being translated.

regards,

Mulyadi.

Double title = "you must read!" :)

Ok, don't worry, we'll read it ASAP :) It was a post in detik.com on May 5th 2009, discussing about police investigation on a murder of a national company's director. It drew so much people's attention during mid April-May, since police has indications that Antasari Azhar, the head of Corruption Eradication Commission, is the mastermind.

Let's see what will the police's conclusion be .... 

Wiranto as presidential candidate?

Notice the first sentence? It is written "Presidential candidate Wiranto" and so on. Hehehe, so he swaps position with Jusuf Kalla? :) The complete post is here for clarification.

kernel panic when calling printk() inside scheduler code...

An interesting question arose in kernelnewbies mailing list. Basically, somebody somebody asked why calling printk() when intercepting scheduler code (using jprobe) results in kernel panic. Full thread can be read here.

My analysis, somehow during module initialization, printk() must not be executed to avoid recursively waking up klogd. The original poster said it didn't help. Finally, he found a workaround, that is by incrementing a variable named oops_in_progress. I bet works like a flag. When it's greater than zero, it prevents klogd to wake up. Problem solved!

Note: klogd is a kernel thread that reads kernel ring buffer and send it to user space daemon syslogd if there's any entries there. Those are the messages that we see ending up in /var/log/messages (for non critical kernel messages).

regards,

Mulyadi.

the way we respond to a question

Probably, you think that's easy. Listen to the question, think for a moment (or longer...if time allows) and explain your answer.

But when I looked at this blog post, I am a bit intrigued. For those who don't know who that guy is, he's Kir Kolyshkin. One of the main developer of OpenVZ, a container solution for Linux. Container is a kind of virtualization which is done in OS level. System calls are intercepted so every guest thinks he owns the whole system. But unlike User Mode Linux, OpenVZ is not using ptrace, but guest kernel is patched so system call is now routed to a "stub" in host kernel. So certainly, Kir is not average developer and far from mediocre level....

This paragraph is interesting:
"I think the talk was well received and I had about 10 different interesting questions, one is puzzling enough so I was not able to provide a good answer. This is definitely a sign of a good audience."

Most of the time, especially when some (doubtly) educated people receive tough question, he will turn into defending mode or attack the questioner back. But Kir thinks differently. He admitted he can not answer all questions, a normal thing for normal human being. And he appreciated that.

I wish this kind of attitute is something we can learn and practice in our everyday life.....

regards,

Mulyadi.

simple function intercept in Linux

Few days ago, during my spare times, I wrote an article for an ezine named Echo zine. The article describe about how one could intercept a library call toward the usual libc library. For example, instead of calling normal rand() function of glibc, it will end up executing your own defined rand(). The trick is to utilize LD_PRELOAD environment variable so your library has higher priority to be linked into the ELF executable.

Without further ado, here's the link:

http://ezine.echo.or.id/ezine20/e20_0x05.txt

I welcome your feedback and comments...

regards,

Mulyadi.

First time I boot my UML guest, I got eth0, then eth1..and so on, why?

It has been itching my head for a while, "why my Slackware UML guest OS gets different eth device name everytime it boots?". It's always incremented e.g first boot I got eth0, later I got eth1 and so on. For note: I didn't build the Slackware by myself, I downloaded from somewhere. Maybe it's uml.harlowill.com or something, sorry I couldn't recall. One thing for sure, it's Slackware 12.0

Previously, I absolutely had no idea why it happened. But lately I was thinking it could be something with udev (after all, it is the one which deals with device file, right?).

Let's boot the UML guest one more time and find out what we could dig. Note that I use the following parameter when I invoke the guest:
eth0=tuntap,tap0

Guest boots up and login prompt shows up. udev should put its configuration under /etc/udev or something alike, so let's dive. Found it, it's in /etc/udev/rules.d/75-network-devices.rules. Here is the snippet that explains my headache:
KERNEL=="eth?", ATTR{address}=="4e:82:25:ae:b1:bb", NAME="eth0"

I see the correlation now. If I don't specificly declare the MAC address of the ethernet device my UML guest will use, UML will pick one randomly. And udev inside the guest will think it's a different ethernet device.

Finally, simple solution. Put that MAC address as part of "eth" parameter:
eth0=tuntap,tap0,4e:82:25:ae:b1:bb

Now I get consistent naming!

PS: Looks like my "research" insting is back. Hopefully, it will stay inside me for long time....

regards,

Mulyadi.