Long story short, currently we see some strange, aggressive ICMPv6 scanning originating from multiple IP addresses inside the Amazon network at my employer Uberspace. The targets are not only used addresses but also internal and non-existent ones. The latter is an issue as it generates a significant amount of neighbor discovery traffic, resulting in increased resource usage. Since we have some other strange issues with our routers, I decided to block and monitor this unwanted traffic, just in case. Therefore, I added some firewall rules like this:
3 chain=prerouting action=jump jump-target=icmpv6_rate_limit in-interface=sfp-sfpplus12 icmp-options=128:0 protocol=icmpv6
11 chain=icmpv6_rate_limit action=accept dst-limit=100/1m,200,src-address/5m
12 chain=icmpv6_rate_limit action=drop log=yes log-prefix="icmpv6_limit_exceeded"
They seemed to work, our monitoring was happy, and the scanning was stopped. At least I thought so. The scanning was indeed stopped, but also the monitoring of some of our customers, which is not so good. First, we thought they may monitor quite aggressive because 100 ICMP echo requests per minute should be enough for all cases. But after checking their reports, we saw that the limiting blocked some requests in a strange, non-deterministic way even when they are far below the 100 requests/min. The big question was WHY?
Let’s start with what I believed the limiting would do. But first some context: we run these filters on Mikrotik hardware, which runs a Linux and the firewall tooling uses the netfilter logic below the surface. I assumed that 100 packets per minute would mean 100 packets per minute, ignoring how they are distributed. Meaning, if I would like to send 99 packets in the first second, this would leave me with 1 packet for the next 59 seconds. Right? Not so much. It’s true that some rate limiting is implemented this way, like the one in haraka (we had some problems with this too), but limiting in netfilter isn’t.
So how does it work in netfilter? First we found a serverfault answer linking to the netfilter limiting docs, but they weren’t helpful, focusing mainly on how the burst limit works, and we knew this wasn’t the issue. But there was another answer below the original question, stating that four packets per hour would mean on toke every 15 minutes. Whatever a toke is. This would mean that our filter would actually create one token every 0.6 seconds. WHAT? But with this information, we found a paper by Nicolas Bouliane from 2007 explaining how the token bucket filter algorithm is or was implemented in the netfilter limit module. Using this knowledge, we really understood what we already assumed.
Instead of ignoring the distribution, our filter above meant that a source could only send a packet every 0.6 seconds. So it depends a lot on the timing of the source sending the requests (if it should send one request per second), whether they are blocked or not. That would have all been below the limit if the distribution wouldn’t matter. Another example would be a limit of now 240 packets per minute, meaning one packet every 0.25 seconds or every 1/240th of a minute. The reason to implement it this way most probably is that the distribution-independent version would need netfilter to keep records of all packets in the last minute with a timestamp, checking them regularly if they are now out of the moving window. This wouldn’t be that performant compared to the actually implemented version if we want to push packets at high speeds.
So the solution to our problem was to rethink how limiting works and find a limit that wouldn’t accidentally match a monitoring tool sending its requests at slightly below 1 request per second.