20 December 2009

12 December 2009

Contributing to laptop mode tools

As confirmed by laptop mode tools' changelog, my patch is finally merged into their core code. I simply contribute few line to show statistics regarding write and read frequency of every programs recorded during lm-profiler's runtime.

So, how the code looks like?

—- lm-profiler.old2009-06-03 21:12:52.000000000 +0700

+++ lm-profiler2009-06-26 21:08:54.000000000 +0700

@ -209,6 +209,11 @


done
printf ’\r \r’
stop_profiling

+

+echo “Write frequency : ”; cat $WORKDIR/write_accesses_* | sed -e ’s/[ \t]*//;s/[ \t]*$//’ -e ’/$/ d’ | sort | uniq -c | sort -n

+echo “Read frequency : ”; cat $WORKDIR/read_accesses_* | sed -e ’s/[ \t]*//;s/[ \t]*$//’ -e ’/$/ d’ | sort | uniq -c | sort -n

+echo;

+
NETPROFILE=`profilenet`
echo “Profiling run completed.”

Looks awful? I admit it. In short, it grabs files created by lm-profiler during its run time, trims out the blanks and then sort them, while at the same time showing their frequencies ascendingly.

NB: Actually it's for my own reminder, but to share with you all. To easily convert text to HTML (and dealing with all those escape characters etc), you could use http://www.textism.com/tools/textile/index.php. Simply paste your text there and click the button, voila..you get the HTML-ized text!

regards,

Mulyadi

25 November 2009

Why so serious? :D

Got this quote from kernelnewbies mailing list....hehehehheheheh :D

"What happens when you read some doc and either it doesn't answer your question or is demonstrably wrong? In Linux, you say "Linux sucks" and go read the code. In Windows/Oracle/etc you say "Windows sucks" and start banging your head  against the wall."

-Denis Vlasenko on lkml

PS: s/banging your head*wall/take it for granted/g is better I guess :D I believe many Linux users also bang his/her head when learning Linux for the first time. :))

Moral: just don't bang your head too hard, it hurts, you know? :D

09 November 2009

how to catch white space(s) using grep?

My definition: White space characters are anything that appears as "blank" a.k.a nothing in screen. They include tab, space, carriage return and so on.

As you know, grep provides a way to catch certain characters class or range. Specifically for white spaces, you can use [[:space:]] or [[:blank:]]. Notice the double [[ and ]] !!!

So, suppose you have text file named test.txt that contains:

     hehe /var/www/
hehe /var/www2/
ttt hehe /var/www3/
     heho /var/www/

Executing:

$ grep ^hehe test.txt

will yield:

hehe /var/www2/

but this:

$ grep -E '^[[:blank:]]*hehe' test.txt

yields:

       hehe /var/www/
hehe /var/www2/

In human words, '[[:blank:]]*' will catch zero or more appearance of space or tab before the word "hehe". If you want to catch at least single appearance of any of them, use "+" instead. Oh and let me remind you again, use -E so that "+" doesn't lose its special meaning.

Note: initially, i thought i simply use [:space:] or [:blank:] and end up in something-is-wrong-but-I-dont-know-what land. Turns out, I didn't read the man page carefully (poor me). Since they are built-in classes, I still need to enclose them with another "[" and "]". Valuable experience.....

regards,

Mulyadi.

15 October 2009

Misunderstanding of rate limit concept in iptables

One day, I decided to learn more about iptables. This time, I found "limit" feature quite interesting to try. An excerpt from iptables manual page:

"limit

This module matches at a limited rate using a token bucket filter. A
rule using this extension will match until this limit is reached
(unless the ‘!’ flag is used). It can be used in combination with the
LOG target to give limited logging, for example."

OK, can't wait to get my hand dirty on it. I fired up my VM (Virtual Machine) guest and type this in guest's console:
# iptables -A INPUT -i eth0 -m limit -m icmp --icmp-type echo-request --limit 1/min -j RETURN

To avoid confusion, I assume the default policy of INPUT chain is ACCEPT. Further, there is no other rule in INPUT chain other than what I typed above.

What is the above rule supposed to do? My understanding, at that point, that it will rate limit the ICMP echo request packets up to 1 packet per minute. Thus, only 1 packet during 1 minute interval will be processed. Further packets will be queued in memory awaiting to be processed. My intention to try this feature is simply to find idea to prevent DDoS, but my "assumption of queueing" made me think that this is not really safe. If there 1 million packets waiting to be processed, eventually your machine's memory will be exhausted, no?

But OK, let's put aside that fearness. I flood ping my VM guest (ping -f, if you don't know how to do it. You have to be root to do this). But guess what? 100% of all ICMP packets are responded really fast!!! What's wrong?

Then I did various test. Replace RETURN target with DROP, not using -i, not specificly rate limiting echo-request, etc. Nothing works! tcpdump still showed me that there were lots of echo request - echo reply packets flowing back and forth between my host and my VM guest.

I almost concluded that "limit" was not working as I thought. Perhaps this is a job of iproute tools, something like we do to rate limit packets using CBQ, HTB etc.

But then I smell something fishy:
# iptables -L -n -v | head

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)



pkts bytes target prot opt in out source destination
0 0 RETURN icmp -- * * 0.0.0.0/0 0.0.0.0/0
limit: avg 1/min burst 5 icmp type 8


Notice the "pkts" field? It's a counter that denotes how many packets are entering certain rule. Also notice there is global counter displayed in INPUT chain (inside the bracket).

So, what's special with them? When I repeat my flood ping test, I saw both the limit rule and the global INPUT counter increased! Thus, something is wrong in my assumption. If indeed the packets were successfully rate limited, at least ACCEPT counter won't be increased as fast as rate limit counter grew.

The answer? Back on manual page. Looks like my English skill was really tested this time. "A rule using this extension will match until this limit is reached". Uhuh...I see...

Confused? Let me explain it as simple as I can:
Assume you use 1 packet per second as limit. During the first minute interval, if a packet arrives, it will hit INPUT chain and checked against rate limit rule. Does it match? Of course! It is still not beyond our limit, right? How about the 2nd, 3rd, 4th and the 5th? They will match too. Why? Because by default, there is burst limit. It will allow several initial packets to get a match, but not all. The default is 5.

What about the rest? For sure, they won't match our limit rule. Again, why? because the limit has been reached (as stated by manual page), thus the limit rule is passed and netfilter will check the next rule. And since there is no more rule in our scenario and the default is to accept in INPUT chain, then all ICMP request packets are accepted and replied!

Solution? Simple. Since the excessive packets will pass our limit rule, then we need to block them right at the next rule e.g:
# iptables -A INPUT -p icmp -m icmp --icmp-type echo-request -j DROP

Voila! We successfully rate limit the ICMP! Woohoo! Case closed.... phewwww

Lesson taken: do not underestimate manual page. Read it very very carefully and make sure you understand every word in it. Misunderstanding of the meaning even a single word could lead to significant difference between successful or stressful trial-and-error implementation. You've been warned...

regards,

Mulyadi

01 October 2009

A little patch that made into main Linux kernel git repository

Commit-ID: 1ad0560e8cdb6d5b381220dc2da187691b5ce124


Gitweb: http://git.kernel.org/tip/1ad0560e8cdb6d5b381220dc2da187691b5ce124
Author: Mulyadi Santosa <mulyadi.santosa@gmail.com>

AuthorDate: Sat, 26 Sep 2009 02:01:41 +0700
Committer: Ingo Molnar <mingo@elte.hu>
CommitDate: Thu, 1 Oct 2009 10:12:03 +0200


perf tools: Run generate-cmdlist.sh properly


Right now generate-cmdlist.sh is not executable, so we
should call it as an argument ".".


This fixes cases where due to different umask defaults
the generate-cmdlist.sh script is not executable in
a kernel tree checkout.


Signed-off-by: Mulyadi Santosa <mulyadi.santosa@gmail.com>
Acked-by: Sam Ravnborg <sam@ravnborg.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Mike Galbraith <efault@gmx.de>

Cc: Paul Mackerras <paulus@samba.org>

Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
LKML-Reference: <f284c33d0909251201w422e9687x8cd3a784e85adf7d@mail.gmail.com>
Signed-off-by: Ingo Molnar <mingo@elte.hu>


---

tools/perf/Makefile | 2 +-

1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/tools/perf/Makefile b/tools/perf/Makefile
index b5f1953..5881943 100644

--- a/tools/perf/Makefile

+++ b/tools/perf/Makefile

@@ -728,7 +728,7 @@ $(BUILT_INS): perf$X
common-cmds.h: util/generate-cmdlist.sh command-list.txt

common-cmds.h: $(wildcard Documentation/perf-*.txt)


- $(QUIET_GEN)util/generate-cmdlist.sh > $@+ && mv $@+ $@
+ $(QUIET_GEN). util/generate-cmdlist.sh > $@+ && mv $@+ $@


$(patsubst %.sh,%,$(SCRIPT_SH)) : % : %.sh

$(QUIET_GEN)$(RM) $@ $@+ && \

15 July 2009

Langkah pertamaku di cyberspace.....

pagi ini buka web detik.com, ternyata ada pengumuman lomba nge-blog. So, kenapa tidak saya coba? Itung-itung berbagi pengalaman dengan sesama netters. 

Pertama kali ngenet sekitar tahun 1996. Waktu itu yang namanya internet bagai barang mewah bagi masyarakat biasa seperti saya. Jumlah ISP (Internet service provider) juga masih bisa dihitung dengan jari, itupun terpusat di Jakarta atau Surabaya. Kecepatan koneksi? Hehehehe, seingat saya waktu itu modem kebanyakan masih 9600 atau 14400 bps :D Bayangkan saja seberapa lambatnya jika koneksi seperti itu sekarang digunakan untuk ngebrowse web dengan content canggih semacam flash dan berisi beberapa banner plus teknologi AJAX. Bisa-bisa kelenger :)

Jadi kesan pertama waktu itu, berinternet itu menyenangkan tapi juga butuh kesabaran. Ya sabar karena koneksinya sendiri lambat, juga kalau line teleponnya putus. Whahahha, ini yang kadang bikin dongkol. Enak-enak download, telepon masuk, putuslah sudah koneksi. Waktu itu program semacam GetRight belum saya ketahui, jadi ya udah....ulang lagi.

Oh iya lupa, sebenernya saya ini sedikit "teracuni" kakak sepupu saya. Seingat saya dulu punya bisnis BBS (Bulletin Board System) di awal tahun 1990-an. Jangan tanya saya apa itu BBS, yang saya tahu, waktu itu saya diperkenalkan ama semacam layar chatting. Saya sendiri waktu itu coba-coba chat dengan kenalan kakak sepupu. Dari situ saya mikir "wah, hebat juga BBS. mungkin suatu saat gak cuma tulisan, tapi juga suara dan video bisa lewat koneksi semacam ini".  Sekarang kita lihat ini sudah jadi kenyataan. VoIP sudah mulai merakyat, video conference berbasis Internet juga banyak dipakai baik oleh perusahaan, pemerintah dan personal.

Begitu lihat layar browser, apa yang dicari? Untungnya search engine seperti Yahoo sudah exist, jadi yang dicari website soal game. Loh? Maklum pecandu game (bahkan sampai sekarang). Nyari move list karakter Mortal Kombat contohnya :)) Ya gimana lagi, waktu itu informasi semacam ini adanya di majalah game luar negeri, tapi gak kebeli. Jadi yang rada murah, ngenet dompleng fasilitas kampus. Tapi yang namanya fasilitas umum, antre nya luar biasa, hehehheheh. Jangan heran kalau kadang yang keluar cuma title halaman terus time out. Jadi ya retry lagi retry lagi.

Kalau soal chat, so pasti dicoba. Seingat saya waktu itu yang populer duluan IRC, baru Yahoo messenger. Apa hayo yang terkenal di IRC? Yup, channel #bawel di DALnet!!! :) Byeuhhhh, tuh channel bener-bener isinya orang yang kayaknya calon MC. Ada aja yang "diomelin". Belum lagi iklan dan quiz yang hilir mudik. Yang mojok? Jangan tanya :)))) Pasti ada, tapi soal ketemunya beda lawan jenis atau ternyata kena yang aspal (asli tapi palsu) ya mana tahu, wong namanya juga dunia maya

OK guys, that's my personal share today. Wanna share yours?

20 June 2009

How to reduce CPU soft lock up in KVM guest

Ever saw something like below messages inside your KVM (Kernel Virtual Machine) guest's console?

" BUG: soft lockup - CPU#0 stuck for 10s! [swapper:0] "

I did and I find it a bit annoying. If you're inside graphical desktop like environment like KDE or GNOME, you  probably won't notice it directly. But you will likely suffer the same condition, the guest OS somehow become unresponsive for a few moment. In my case, it manifest into stalled CD/DVD access and "ruins" the console display. I had to press Enter few times before I could get back to normal shell prompt. Before I go further, FYI I use Fedora 9, kernel version 2.6.27.23-xx.x.xx.fc9.i686 on a Core Duo powered laptop.

First, why the kernel shows such message? I use the default CentOS 5.3 kernel, so I check the related kernel config inside /boot directory and here is the related configuration item:

CONFIG_DETECT_SOFTLOCKUP=y

What does it do? Ingo Molnar, the writer of this lockup detection patch describe it as:

"From: Ingo Molnar

This patch adds a new kernel debug feature: CONFIG_DETECT_SOFTLOCKUP.

When enabled then per-CPU watchdog threads are started, which try to run once per second. If they get delayed for more than 10 seconds then a callback from the timer interrupt detects this condition and prints out a warning message and a stack dump (once per lockup incident). The feature is otherwise non-intrusive, it doesnt try to unlock the box in any way, it only gets the debug info out, automatically, and on all CPUs affected by the lockup.

Tested this on x86, both with the feature enabled (in which case a provoked lockup was correctly detected) and with the feature disabled. It is CPU-hotplug aware. Should work on every architecture. "

Pay attention here that the watchdog mentioned in the above description has nothing to do with NMI (Non Maskable Interrupt) watchdog. NMI watchdog deal with hard CPU lockup, while the above mentioned lockup watchdog can't. It's just a kernel thread that will stuck if CPU hangs.

I suspect it might be a bug in KVM driver (or specificly, KVM for Intel VT in my case). I came to this hypothesis because the help section of soft lockup patch says:

"Say Y here to enable the kernel to detect "soft lockups", which are bugs that cause the kernel to loop in kernel mode for more than 10 seconds, without giving other tasks a chance to run."

Great...so IMO KVM is too busy on something, or.... something is delaying KVM guest to run. I almost rush to compile my own kernel image using full preemption, hoping that it could squash the problem. But I was tempted to Google a bit more. Interesting result, a post in a mailing list (I forgot which one) suggest to set the CPU frequency into static. Let's try, I edited /etc/sysconfig/cpuspeed so the related lines become:

MAX_SPEED=1333000
MIN_SPEED=1333000

I pick that frequency because it's the middle number between the three available frequencies: 1833000, 1333000 and 1000000 Hz. So theoritically I still get adequate computing power to most job without draining the battery too soon.

Execute:

# service cpuspeed restart

Make sure it's applied correctly:

# grep '1333000' -r /sys/devices/system/cpu/ 
/sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq:1333000
/sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq:1333000

...
/sys/devices/system/cpu/cpu1/cpufreq/scaling_min_freq:1333000
/sys/devices/system/cpu/cpu1/cpufreq/scaling_max_freq:1333000

...

Then I ran my KVM guest again. I did few tasks in it, let it went idle, repeat and so on. During my test for about an hour, the result was promising! It became stable. Well, few lock up happened though, but it was far reduced. Thing that I notice is lockup also happens when I switch into another virtual desktop or if fairly heavy swapping in/out is on the way. So, to further reduce lock-ups, I avoid switching to another virtual desktop (ok, that sucks, but I could live with that) and close any unneccesary applications to conserve virtual memory as much as I can.

Why it works? All I can say is by making the frequency static, you also avoid timer interrupt delivery frequency being changed too. It stabilize the kernel timing and also indirectly stabilize the KVM guest timing. Previously, I was using ondemand power governor and as you might be aware of, it adapts the CPU frequency according to the load quite aggresively. So, frequency was juggling between all three available frequencies. Conservative governor didn't lend a help here. Pity, I assume KVM still force conservative governor to switch to highest frequency then drop most of the time because most of the codes could be run natively instead of being translated.

regards,

Mulyadi.

24 May 2009

Double title = "you must read!" :)

Ok, don't worry, we'll read it ASAP :) It was a post in detik.com on May 5th 2009, discussing about police investigation on a murder of a national company's director. It drew so much people's attention during mid April-May, since police has indications that Antasari Azhar, the head of Corruption Eradication Commission, is the mastermind.

Let's see what will the police's conclusion be .... 

Wiranto as presidential candidate?

Notice the first sentence? It is written "Presidential candidate Wiranto" and so on. Hehehe, so he swaps position with Jusuf Kalla? :) The complete post is here for clarification.

09 March 2009

kernel panic when calling printk() inside scheduler code...

An interesting question arose in kernelnewbies mailing list. Basically, somebody somebody asked why calling printk() when intercepting scheduler code (using jprobe) results in kernel panic. Full thread can be read here.

My analysis, somehow during module initialization, printk() must not be executed to avoid recursively waking up klogd. The original poster said it didn't help. Finally, he found a workaround, that is by incrementing a variable named oops_in_progress. I bet works like a flag. When it's greater than zero, it prevents klogd to wake up. Problem solved!

Note: klogd is a kernel thread that reads kernel ring buffer and send it to user space daemon syslogd if there's any entries there. Those are the messages that we see ending up in /var/log/messages (for non critical kernel messages).

regards,

Mulyadi.

08 March 2009

the way we respond to a question

Probably, you think that's easy. Listen to the question, think for a moment (or longer...if time allows) and explain your answer.

But when I looked at this blog post, I am a bit intrigued. For those who don't know who that guy is, he's Kir Kolyshkin. One of the main developer of OpenVZ, a container solution for Linux. Container is a kind of virtualization which is done in OS level. System calls are intercepted so every guest thinks he owns the whole system. But unlike User Mode Linux, OpenVZ is not using ptrace, but guest kernel is patched so system call is now routed to a "stub" in host kernel. So certainly, Kir is not average developer and far from mediocre level....

This paragraph is interesting:
"I think the talk was well received and I had about 10 different interesting questions, one is puzzling enough so I was not able to provide a good answer. This is definitely a sign of a good audience."

Most of the time, especially when some (doubtly) educated people receive tough question, he will turn into defending mode or attack the questioner back. But Kir thinks differently. He admitted he can not answer all questions, a normal thing for normal human being. And he appreciated that.

I wish this kind of attitute is something we can learn and practice in our everyday life.....

regards,

Mulyadi.

21 February 2009

simple function intercept in Linux

Few days ago, during my spare times, I wrote an article for an ezine named Echo zine. The article describe about how one could intercept a library call toward the usual libc library. For example, instead of calling normal rand() function of glibc, it will end up executing your own defined rand(). The trick is to utilize LD_PRELOAD environment variable so your library has higher priority to be linked into the ELF executable.

Without further ado, here's the link:

http://ezine.echo.or.id/ezine20/e20_0x05.txt

I welcome your feedback and comments...

regards,

Mulyadi.

18 January 2009

First time I boot my UML guest, I got eth0, then eth1..and so on, why?

It has been itching my head for a while, "why my Slackware UML guest OS gets different eth device name everytime it boots?". It's always incremented e.g first boot I got eth0, later I got eth1 and so on. For note: I didn't build the Slackware by myself, I downloaded from somewhere. Maybe it's uml.harlowill.com or something, sorry I couldn't recall. One thing for sure, it's Slackware 12.0

Previously, I absolutely had no idea why it happened. But lately I was thinking it could be something with udev (after all, it is the one which deals with device file, right?).

Let's boot the UML guest one more time and find out what we could dig. Note that I use the following parameter when I invoke the guest:
eth0=tuntap,tap0

Guest boots up and login prompt shows up. udev should put its configuration under /etc/udev or something alike, so let's dive. Found it, it's in /etc/udev/rules.d/75-network-devices.rules. Here is the snippet that explains my headache:
KERNEL=="eth?", ATTR{address}=="4e:82:25:ae:b1:bb", NAME="eth0"

I see the correlation now. If I don't specificly declare the MAC address of the ethernet device my UML guest will use, UML will pick one randomly. And udev inside the guest will think it's a different ethernet device.

Finally, simple solution. Put that MAC address as part of "eth" parameter:
eth0=tuntap,tap0,4e:82:25:ae:b1:bb

Now I get consistent naming!

PS: Looks like my "research" insting is back. Hopefully, it will stay inside me for long time....

regards,

Mulyadi.

How to execute multiple commands directly as ssh argument?

 Perhaps sometimes you need to do this: ssh user@10.1.2.3 ls It is easy understand the above: run ls after getting into 10.1.2.3 via ssh. Pi...