Enhancing Inline IPS Performance with Kernel String Matching
21 April, 2007
The iptables QUEUE and NFQUEUE targets are used by both open source and commercial
Intrusion Prevention Systems. These targets allow a userspace process (the IPS in
this case) to acquire packet data from the Linux kernel via a netlink socket and set
verdicts on whether packets should be forwarded only after the IPS has sent the packets
through its detection engine. This can provide a effective means from protecting
other systems from attack (subject to the usual concerns over false positives).
Using iptables provides a flexible means for placing a QUEUE rule within an existing iptables policy. For example, if you are not running a DNS server behind an iptables firewall, then you can use the DROP target against any attempts to communicate with a DNS server before sending such packets to a userspace IPS - which would needlessly place additional load on the CPU when such traffic can just be dropped from within the kernel directly before it ever hits the QUEUE rule:
# iptables -A FORWARD -p udp --dport 53 -j DROP
# iptables -A FORWARD -j QUEUE
This is a nice consequence, but is only possible in this case because there is no
internal DNS server to attack in the first place. What about using facilities
provided by iptables to reduce the packet load that is queued to a userspace IPS
for servers that do exist? Say, for a web server?
An IPS such as snort_inline can be deployed with a large signature set, and most Snort signatures use the content or uricontent keywords to search packet application layer data for malicious content. In the context of snort_inline, usually the QUEUE or NFQUEUE rule is the default rule in the FORWARD chain - this ensures that snort_inline inspects all packets before they are forwarded through the system. So, you could say the iptables policy has a "default QUEUE stance".
However, by using the iptables string match extension and an iptables policy built by fwsnort, we can change the "default QUEUE stance" to "only QUEUE packets that match a Snort content field". The resulting performance gain can be substantial as we will see below. Fwsnort accomplishes this by parsing the Snort signature set that is deployed in snort_inline, and then building an iptables policy that makes heavy use of the string match extension to ACCEPT all packets in the FORWARD chain that do not match any of the content fields in the signature set, and QUEUE those that do. In general, this greatly reduces the number of packets that have to be queued to snort_inline since (usually) most traffic is not malicious.
As a simple example, consider the following Snort signature:
alert tcp any any -> any any (msg:"fwsnort download"; content:"fwsnort/download";
classtype:web-application-attack; sid:12325678; rev:1;)
This signature is for illustration purposes only and is written just to force
snort_inline to inspect all TCP packets for the string "fwsnort/download". With
this signature deployed in snort_inline on one of my systems (an old PIII with 100Mb
Ethernet interfaces running the 2.6.20.3 kernel), I measured the following
throughput with netperf.
This test used a default QUEUE rule in iptables, so every TCP packet is queued to
snort_inline:
# netperf -v 2 -L 192.168.50.1 -H 192.168.50.2 TCP STREAM TEST from 192.168.50.1 (192.168.50.1) port 0 AF_INET \ to 192.168.50.2 (192.168.50.2) port 0 AF_INET Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 87380 16384 16384 10.02 60.07 Alignment Offset Bytes Bytes Sends Bytes Recvs Local Remote Local Remote Xfered Per Per Send Recv Send Recv Send (avg) Recv (avg) 8 8 0 0 7.524e+07 16384.33 4592 1444.75 52076 Maximum Segment Size (bytes) 1448The throughput number above indicates that netperf was able to achieve 60.07 Mb/sec over the life of the 10 second test. Now, let's use fwsnort to convert the Snort signature into an equivalent iptables rule, and deploy it:
# fwsnort --snort-sid 12325678 --QUEUE
[+] Parsing Snort rules files...
[+] Found sid: 12325678 in test.rules
Successful translation.
[+] Logfile: /var/log/fwsnort.log
[+] Snort rule set directory for rules to be queued to userspace:
/etc/fwsnort/snort_rules_queue
[+] iptables script: /etc/fwsnort/fwsnort.sh
# /etc/fwsnort/fwsnort.sh
[+] Adding test rules.
Rules added: 2
[+] Finished.
Now, executing the netperf throughput test generates the following result:
# netperf -v 2 -L 192.168.50.1 -H 192.168.50.2
TCP STREAM TEST from 192.168.50.1 (192.168.50.1) port 0 AF_INET \
to 192.168.50.2 (192.168.50.2) port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
87380 16384 16384 10.01 94.11
Alignment Offset Bytes Bytes Sends Bytes Recvs
Local Remote Local Remote Xfered Per Per
Send Recv Send Recv Send (avg) Recv (avg)
8 8 0 0 1.178e+08 16384.74 7189 1440.01 81798
Maximum
Segment
Size (bytes)
1448
So, in this test we achieved 94.11 Mb/sec sustained throughput - that's about a 57%
speed increase!
This may sound good, but as always there are some tradeoffs. First, because this strategy causes iptables to only selectively queue packets to userspace, snort_inline does not have the opportunity inspect an entire stream. This implies that the stream preprocessor needs to be disabled since otherwise signatures with the flow keyword will never match. Secondly, because the preliminary signature match is performed by iptables, taking into account application layer encodings (such as URL encoding used in web servers) is not possible. This results in a reduced ability to detect attacks. In some cases however, performance requirements may override these considerations. Also, some attacks do not need to bother with attempts at obfuscation because the target environment is so rich.
Significant work needs to be performed to validate this as a viable strategy for increasing IPS performance, but these preliminary results are encouraging. On a final note, this strategy is applicable to any IPS that makes use of the iptables queuing mechanism - not just snort_inline.
Also, thanks are in order to Hank Leininger of KoreLogic Security for suggesting the combination of the QUEUE target and string matching as a way to speed up inline Snort implementations at a talk I gave about Linux Firewalls at ShmooCon 2007. I implemented the --QUEUE and --NFQUEUE targets in fwsnort-1.0 as a result of meditating on this for a while.
