15 October 2009

Misunderstanding of rate limit concept in iptables

One day, I decided to learn more about iptables. This time, I found "limit" feature quite interesting to try. An excerpt from iptables manual page:


This module matches at a limited rate using a token bucket filter. A
rule using this extension will match until this limit is reached
(unless the ‘!’ flag is used). It can be used in combination with the
LOG target to give limited logging, for example."

OK, can't wait to get my hand dirty on it. I fired up my VM (Virtual Machine) guest and type this in guest's console:
# iptables -A INPUT -i eth0 -m limit -m icmp --icmp-type echo-request --limit 1/min -j RETURN

To avoid confusion, I assume the default policy of INPUT chain is ACCEPT. Further, there is no other rule in INPUT chain other than what I typed above.

What is the above rule supposed to do? My understanding, at that point, that it will rate limit the ICMP echo request packets up to 1 packet per minute. Thus, only 1 packet during 1 minute interval will be processed. Further packets will be queued in memory awaiting to be processed. My intention to try this feature is simply to find idea to prevent DDoS, but my "assumption of queueing" made me think that this is not really safe. If there 1 million packets waiting to be processed, eventually your machine's memory will be exhausted, no?

But OK, let's put aside that fearness. I flood ping my VM guest (ping -f, if you don't know how to do it. You have to be root to do this). But guess what? 100% of all ICMP packets are responded really fast!!! What's wrong?

Then I did various test. Replace RETURN target with DROP, not using -i, not specificly rate limiting echo-request, etc. Nothing works! tcpdump still showed me that there were lots of echo request - echo reply packets flowing back and forth between my host and my VM guest.

I almost concluded that "limit" was not working as I thought. Perhaps this is a job of iproute tools, something like we do to rate limit packets using CBQ, HTB etc.

But then I smell something fishy:
# iptables -L -n -v | head

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)

pkts bytes target prot opt in out source destination
0 0 RETURN icmp -- * *
limit: avg 1/min burst 5 icmp type 8

Notice the "pkts" field? It's a counter that denotes how many packets are entering certain rule. Also notice there is global counter displayed in INPUT chain (inside the bracket).

So, what's special with them? When I repeat my flood ping test, I saw both the limit rule and the global INPUT counter increased! Thus, something is wrong in my assumption. If indeed the packets were successfully rate limited, at least ACCEPT counter won't be increased as fast as rate limit counter grew.

The answer? Back on manual page. Looks like my English skill was really tested this time. "A rule using this extension will match until this limit is reached". Uhuh...I see...

Confused? Let me explain it as simple as I can:
Assume you use 1 packet per second as limit. During the first minute interval, if a packet arrives, it will hit INPUT chain and checked against rate limit rule. Does it match? Of course! It is still not beyond our limit, right? How about the 2nd, 3rd, 4th and the 5th? They will match too. Why? Because by default, there is burst limit. It will allow several initial packets to get a match, but not all. The default is 5.

What about the rest? For sure, they won't match our limit rule. Again, why? because the limit has been reached (as stated by manual page), thus the limit rule is passed and netfilter will check the next rule. And since there is no more rule in our scenario and the default is to accept in INPUT chain, then all ICMP request packets are accepted and replied!

Solution? Simple. Since the excessive packets will pass our limit rule, then we need to block them right at the next rule e.g:
# iptables -A INPUT -p icmp -m icmp --icmp-type echo-request -j DROP

Voila! We successfully rate limit the ICMP! Woohoo! Case closed.... phewwww

Lesson taken: do not underestimate manual page. Read it very very carefully and make sure you understand every word in it. Misunderstanding of the meaning even a single word could lead to significant difference between successful or stressful trial-and-error implementation. You've been warned...




lazzybear said...

yup its the burts that count and also the chain

try to make it like this
iptables -A INPUT -i eth0 -m limit -m icmp --icmp-type echo-request --limit 1/min -j DROP

it suppose to be drop right ?

Kenny Hendrick said...

Can someone tell me if, when we limit a block of ip addresses (in my case they were coming from someone out of west palm beach florida ALL DAY), does the offending machine receive a notice that we are limiting them?

I particularly am interested because although I should have addressed the issue a lot earlier since the server was all pepped up running loud but didn't have time to inspect what was happening. Upon reviewing the logs I found that although the ip addresses were changing that the mac address was not...and then after a whois and traceroute and blah blah blah I decided to limit the whole block owned by ymax)....


Did my machine just sling back a slew of messages to that other machine that was attacking me? Immediately the server settled down and it's been nice and smooth all evening .... just a bunch of bots bouncing around in there and not much else but at least it's quiet again....

phew, I seem to be a tad chatty after that last cup of java.....