Michael Rash, Security Researcher

Network Security    [Summary View]

How to avoid ClamAV matches on bundled Snort rules

ClamAV matches on Snort rules Recently a psad user notified me via email that the psad-2.1.7.tar.gz tarball was flagged by ClamAV as being infected with Exploit.HTML.MHTRedir-8. After checking a few things, it turns out that ClamAV is triggering on a Snort rule in the Emerging Threats rule set which is bundled in both psad and fwsnort. The following analysis shows exactly what ClamAV is detecting and why, and also provides some guidance for how to avoid this for any software projects that distribute Snort rules. Similar logic would apply to other software engineering efforts - including commercial intrusion detection systems - that are (by their nature) looking for malicious artifacts on the filesystem or within network traffic.

First, let's download psad-2.1.7.tar.gz and check the gpg signature (just to make sure we're talking about exactly the same file): $ wget --quiet
$ wget --quiet
$ gpg --verify psad-2.1.7.tar.gz.asc
gpg: Signature made Wed 14 Jul 2010 06:01:42 PM EDT using DSA key ID 0D3E7410
gpg: Good signature from "Michael Rash (Signing key for projects) <>"
Ok, now here is what clamscan says about the psad-2.1.7.tar.gz tarball: $ clamscan psad-2.1.7.tar.gz
psad-2.1.7.tar.gz: Exploit.HTML.MHTRedir-8 FOUND

----------- SCAN SUMMARY -----------
Known viruses: 816934
Engine version: 0.96.1
Scanned directories: 0
Scanned files: 1
Infected files: 1
Data scanned: 6.42 MB
Data read: 1.16 MB (ratio 5.55:1)
Time: 7.169 sec (0 m 7 s)
Let's see which file within the psad-2.1.7 tarball matches the Exploit.HTML.MHTRedir-8 signature: $ tar xfz psad-2.1.7.tar.gz
$ clamscan -r -i psad-2.1.7
psad-2.1.7/deps/snort_rules/emerging-all.rules: Exploit.HTML.MHTRedir-8 FOUND

----------- SCAN SUMMARY -----------
Known viruses: 816934
Engine version: 0.96.1
Scanned directories: 41
Scanned files: 405
Infected files: 1
Data scanned: 12.55 MB
Data read: 6.41 MB (ratio 1.96:1)
Time: 8.446 sec (0 m 8 s)
Intuitively, this makes sense. That is, given that ClamAV is out to identify nasty things within files, and given that Snort rules are designed to identify nasty things as they communicate over the network, it stands to reason that there might be some overlap. This overlap is not an indication of something wrong in either the Snort rules or in ClamAV. Now, let's find out specifically which Snort rule within the emerging-all.rules file is triggering the ClamAV match. We first take a look at the Exploit.HTML.MHTRedir-8 signature: $ cp /var/lib/clamav/main.cvd .
$ sigtool --unpack main.cvd
$ grep Exploit.HTML.MHTRedir-8 main.ndb
The last line above is the entire ClamAV signature, and the pattern 6d68746d6c3a66696c653a2f2f is the key. The ":3:" part identifies the signature as type "normalized HTML", so ClamAV matches the pattern 6d68746d6c3a66696c653a2f2f against the "normalized HTML" representation of each processed file. We can decode the pattern as follows: echo 6d68746d6c3a66696c653a2f2f | xxd -r -p
So, within the emerging-all.rules file, we are interested in any Snort rule that contains the string mhtml:file://. There is also the "{1-20}2168" criteria which says to match the hex bytes 2168 anywhere from 1 to 20 bytes after the first pattern match. $ grep mhtml psad-2.1.7/deps/snort_rules/emerging-all.rules
alert tcp $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS (msg:"ET MALWARE Bundleware Spyware CHM Download"; flow: to_server,established; content:"Referer\: ms-its\:mhtml\:file\://C\:counter.mht!http\://"; nocase; content:"/counter/HELP3.CHM\:\:/help.htm"; nocase; classtype: trojan-activity; sid: 2001452; rev:4;)
Sure enough, sid:2001452 "ET MALWARE Bundleware Spyware CHM Download" has the keyword content:"Referer\: ms-its\:mhtml\:file\://C\:counter.mht!http\://". Even though there are escaping backslashes, the normalized HTML processing in ClamAV takes this into account and matches the pattern anyway from the ClamAV signature.

So, how can we keep the original Snort rule, but change it so that ClamAV not longer flags it?

Fortunately, ClamAV does not interpret the Snort rules convention of specifying non-printable bytes between "|" characters within content fields, so we simply need to change one of characters to hex notation. Snort will still offer the same detection if network traffic matches the rule, and ClamAV won't flag it. So, let's just change the "m" in "mhtml\:file\://" to its hex equivalent, like so: "|6d|html\:file\://". Once we make this change and save the psad-2.1.7/deps/snort_rules/emerging-all.rules file, we rerun clamscan: $ clamscan -r -i psad-2.1.7

----------- SCAN SUMMARY -----------
Known viruses: 816934
Engine version: 0.96.1
Scanned directories: 41
Scanned files: 405
Infected files: 0
Data scanned: 17.23 MB
Data read: 6.41 MB (ratio 2.69:1)
Time: 10.024 sec (0 m 10 s)
That's better. Over the next few days I'll re-release all affected versions of psad and fwsnort with the above change to ensure that there aren't any additional ClamAV matches.

In conclusion, if you are involved in any software engineering effort that distributes or makes use of Snort rules, it is probably a good idea to run distribution packages through ClamAV and see if there are any matches. If so, it may be possible to take advantage of Snort rule syntax options to still achieve the same signature coverage while not having ClamAV flag anything.

Nmap-5.00, Zenmap, and ndiff

Nmap-5.00 Fyodor recently released Nmap-5.00, and this release marks a major milestone in Nmap development as it is now quite mature, has a large following, and is feature rich. With Nmap, service discovery and interrogation has never been easier or more automated, and for me Nmap provided much of the inspiration to develop psad (scan detection via iptables log messages) and fwknop (hide services from all types of scans behind a default-drop packet filter with Single Packet Authorization).

Some of the more interesting Nmap features these days include the Nmap Scripting Engine (NSE), the Zenmap user interface, and ndiff. The NSE is an Nmap extension that allows users to express networking tasks via the Lua embedded programming language, and the resulting scripts are executed via Nmap against targeted systems. As an example of some of the power that NSE provides, a recent update allows Nmap to interrogate a system in order to see if it is infected with the Conficker worm.

Zenmap provides a nice graphical interface to point and click Nmap scanning, complete with interactive editing of Nmap command line arguments, scan results display with context sensitive text colors, and even a network topology viewer to represent scan targets. The screenshot below illustrates the scan results view of a scan against a Netgear router: zenmap scan view

An excellent example of the topology view in Zenmap can be found here.

With the new Nmap release, some questions the security community may ask include:

  • Will scan activity significantly increase? Most likely there will be a burst of scanning over the next few weeks as people adopt and experiment with the new release - especially after the broad news coverage Nmap is getting.
  • By direct examination of network traffic Is it possible to differentiate Nmap-5.00 scans from those that originate from older versions? My guess is most likely not, but a source code diff from older versions should make this clear.
  • Does the new release imply that the Conficker worm will accelerate its decline as more scans are made to find infected systems? Note that Conficker seems to already be on the decline by one measure.
Finally, I wonder if ndiff will change how people use Nmap in the long run? It is great to have accurate scan information about a target, but it is even better to see how this information changes over time. For example, if a system is compromised and is forced to stand up a new backdoor service, then this will cause a "blip" in ndiff results if this system is the target of regular Nmap scans. Or, if a broad policy change is made in a router ACL or firewall rule set, then this can result in broad ndiff changes too. Another example might be if a networked application is upgraded such that it advertises itself differently from one scan to the next (say, via a banner such as "Apache/2.2.11 (Ubuntu) Server at localhost Port 80"), ndiff might alert you more effectively than other techniques (this assumes that you have enabled version scanning).

On a technical note, it is possible to introduce false positives into ndiff output if the Nmap command line is altered from one diff to the next. Suppose that scans for a particular UDP service seem to finish fairly quickly and reliably because the target returns an ICMP port unreachable message (indicating that the service is not filtered). But, in the interest of speeding scans up further, suppose the --max-rtt-timeout argument is used on the Nmap command line, and suppose that timeout is reduced too far. In this case, through no fault of its own, Nmap would report the service as filtered because the ICMP port unreachable message returned after the reduced timer had already expired. If the before and after Nmap scan results are compared, ndiff would report the difference even though the user is responsible for creating it. Nmap is doing its job though, and changing how Nmap is invoked for automated scans is probably not very common. At least, over time the way Nmap is invoked would average out to the same. The main goal of comparing scan results is wonderfully automated by ndiff, and is a powerful mechanism for seeing how network service availability changes over time.

Congratulations to Fyodor and the Nmap developers on a great release.