cipherdyne.org

Michael Rash, Security Researcher



Port Knocking and SPA    [Summary View]

RAM Disks and Saving Your SSD From AFL Fuzzing

The American Fuzzy Lop fuzzer has become a critical tool for finding security vulnerabilities in all sorts of software. It has the ability to send fuzzing data through programs on the order of hundreds of millions of executions per day on a capable system, and can certainly put strain on your hardware and OS. If you are fuzzing a target program with the AFL mode where a file is written and the target binary reads from this file, then AFL is going to conduct a huge number of writes to the local disk. For a solid-state drive this can reduce its life expectancy because of write amplification.

My main workstation is a Mac running OS X Yosemite, and I run a lot of Linux, FreeBSD, and OpenBSD virtual machines under Parallels for development purposes. The drive on this system is an SSD which keeps everything quite fast, but I don't want to prematurely shorten its life through huge AFL fuzzing runs. Normally, I'm using AFL to fuzz fwknop from an Ubuntu-14.10 VM, and what is needed is a way to keep disk writes down. The solution is to use a RAM disk from within the VM.

First, from the Ubuntu VM, let's get set up for AFL fuzzing and show what the disk writes look like without using a RAM disk from the perspective of the host OS X system. This assumes AFL 0.89 has been installed already and is in the current path:
$ git clone https://github.com/mrash/fwknop.git fwknop.git
$ cd fwknop.git
$ ./autogen.sh
$ cd test/afl/
$ ./compile/afl-compile.sh
We're not running AFL yet. Now, from the Mac, launch the Activity Monitor (under Applications > Utilities) and look at current disk utilization: AFL not running disk writes So, not terrible - currently 31 writes per second at the time the snapshot was taken, and that includes OS X itself and the VM at the same time. But, now let's fire up AFL using the digest cache wrapper on the Ubuntu VM (the main AFL text UI is not shown for brevity):
$ ./fuzzing-wrappers/server-digest-cache.sh
[+] All right - fork server is up.
[+] All test cases processed.

[+] Here are some useful stats:

    Test case count : 1 favored, 0 variable, 1 total
           Bitmap range : 727 to 727 bits (average: 727.00 bits)
                   Exec timing : 477 to 477 us (average: 477 us)

[+] All set and ready to roll!
And now let's take a look at disk writes again from OS X: AFL no RAM disk writes Whoa, that's a massive difference - nearly two orders of magnitude. AFL has caused disk writes to spike to over 2,700 per second with total data written averaging at 19.5MB/sec. Long term fuzzing at this level of disk writes would clearly present a problem for the SSD - AFL frequently needs to be left running for days on end in order to be thorough. So, let's switch everything over to use a RAM disk on the Ubuntu VM instead and see if that reduces disk writes:
# mkdir /tmp/afl-ramdisk && chmod 777 /tmp/afl-ramdisk
# mount -t tmpfs -o size=512M tmpfs /tmp/afl-ramdisk
$ mv fwknop.git /tmp/afl-ramdisk
$ cd /tmp/afl-ramdisk/fwknop.git/test/afl/
$ ./fuzzing-wrappers/server-digest-cache.sh
Here is disk utilization once again from the Mac: AFL RAM disk writes We're back to less than 10 writes per second to the SSD even though AFL is going strong on the Ubuntu VM (not shown). The writes for the previous fuzzing run are still shown to the left of the graph (since they haven't quite aged out yet when the screenshot was taken), and new writes are so low they don't even make it above the X-axis at this scale. Although the total execs per second - about 2,000 - achieved by AFL is not appreciably faster under the RAM disk, the main benefit is that my SSD will last a lot longer. For those that don't run AFL underneath a VM, a similar strategy should still apply on the main OS. Assuming enough RAM is available for whatever software you want to fuzz, just create a RAM disk and run everything from it and extend the life of your hard drive in the process.

Software Release: fwknop-2.6.4

fwknop-2.6.4 software release The 2.6.4 release of fwknop is available for download. New functionality has been developed for 2.6.4, including a new UDP listener mode to remove libpcap as a dependency for fwknopd, support for firewalld on recent versions of Fedora, RHEL, and Centos (contributed by Gerry Reno), and support for Michal Zalewski's 'American Fuzzy Lop' fuzzer. Further, on systems where execvpe() is available, all system() and popen() calls have been replaced so that the shell is not invoked and no environment is used. As usual, fwknop has a Coverity Scan score of zero, and the code coverage report achieved by the 2.6.4 test suite is available here.

Here is the complete ChangeLog for fwknop-2.6.4:

  • [server] Added a UDP server mode so that SPA packets can be acquired via UDP directly without having to use libpcap. This is an optional feature since it opens a UDP port (and therefore requires the local firewall be opened for communications to this port), but fwknopd is careful to never send anything back to a client that sends data to this port. So, from the perspective of an attacker or scanner, fwknopd remains invisible. This feature is enabled in fwknopd either with a new command line argument --udp-server or in the fwknopd.conf file with the ENABLE_UDP_SERVER variable. When deployed in this mode, it is advisable to recompile fwknop beforehand with './configure --enable-udp-server' so that fwknopd does not link against libpcap.
  • [server] Replaced all popen() and system() calls with execvpe() with no usage of the environment. This is a defensive measure to not make use of the shell for firewall command execution, and is supported on systems where execvpe() is available.
  • (Gerry Reno) Added support for firewalld to the fwknopd daemon on RHEL 7 and CentOS 7. This is implemented using the current firewalld '--direct --passthrough' capability which accepts raw iptables commands. More information on firewalld can be found here: https://fedoraproject.org/wiki/FirewallD
  • [server] Added support for the 'American Fuzzy Lop' (AFL) fuzzer from Michal Zalewski. This requires that fwknop is compiled with the '--enable-afl-fuzzing' argument to the configure script as this allows encryption/digest short circuiting in a manner necessary for AFL to function properly. The benefit of this strategy is that AFL can fuzz the SPA packet decoding routines implemented by libfko. See the test/afl/ directory for some automation around AFL fuzzing.
  • (Bill Stubbs) submitted a patch to fix a bug where fwknopd could not handle Ethernet frames that include the Frame Check Sequence (FCS) header. This header is four bytes long, and is placed at the end of each Ethernet frame. Normally the FCS header is not visible to libpcap, but some card/driver combinations result in it being included. Bill noticed this on the following platform: BeagleBone Black rev C running 3.8.13-bone50 #1 SMP Tue May 13 13:24:52 UTC 2014 armv7l GNU/Linux
  • [client] Bug fix to ensure that a User-Agent string can be specified when the fwknop client uses wget via SSL to resolve the external IP address. This closes issue #134 on github reported by Barry Allard. The fwknop client now uses the wget '-U' option to specify the User-Agent string with a default of "Fwknop/<version>". In addition, a new command line argument "--use-wget-user-agent" to allow the default wget User-Agent string to apply instead.
  • [python module] When an HMAC key is passed to spa_data_final() then default to HMAC SHA256 if no HMAC mode was specified.

Software Release: fwknop-2.6.3

fwknop-2.6.3 software release The 2.6.3 release of fwknop is available for download. The emphasis in this release is maximizing code coverage through a new python SPA packet fuzzer, and also on fault injection testing with the excellent fault injection library libfiu developed by Alberto Bertogli. Another important change in 2.6.3 is all IP resolution lookups in '-R' mode now happen over SSL to make it harder for an adversary to mount a MITM attack on the resolution lookup. As always, manually specifying the IP to allow through the remote firewall is safer than relying on any network communication - even when SSL would be involved.

Here is the complete ChangeLog for fwknop-2.6.3:

  • [client] External IP resolution via '-R' (or '--resolve-ip-http') is now done via SSL by default. The IP resolution URL is now 'https://www.cipherdyne.org/cgi-gin/myip', and a warning is generated in '-R' mode whenever a non-HTTPS URL is specified (it is safer just to use the default). The fwknop client leverages 'wget' for this operation since that is cleaner than having fwknop link against an SSL library.
  • Integrated the 'libfiu' fault injection library available from http://blitiri.com.ar/p/libfiu/ This feature is disabled by default, and requires the --enable-libfiu-support argument to the 'configure' script in order to enable it. With fwknop compiled against libfiu, fault injections are done at various locations within the fwknop sources and the test suite verifies that the faults are properly handled at run time via test/fko-wrapper/fko_fault_injection.c. In addition, the libfiu tool 'fiu-run' is used against the fwknop binaries to ensure they handle faults that libfiu introduces into libc functions. For example, fiu-run can force malloc() to fail even without huge memory pressure on the local system, and the test suite ensures the fwknop binaries properly handle this.
  • [test suite] Integrated a new python fuzzer for fwknop SPA packets (see test/spa_fuzzing.py). This greatly extends the ability of the test suite to validate libfko operations since SPA fuzzing packets are sent through libfko routines directly (independently of encryption and authentication) with a special 'configure' option --enable-fuzzing-interfaces. The python fuzzer generates over 300K SPA packets, and when used by the test suite consumes about 400MB of disk. For reference, to use both the libfiu fault injection feature mentioned above and the python fuzzer, use the --enable-complete option to the test suite.
  • [test suite] With the libfiu fault injection support and the new python fuzzer, automated testing of fwknop achieves 99.7% function coverage and 90.2% line coverage as determined by 'gcov'. The full report may be viewed here: http://www.cipherdyne.org/fwknop/lcov-results/
  • [server] Add a new GPG_FINGERPRINT_ID variable to the access.conf file so that full GnuPG fingerprints can be required for incoming SPA packets in addition to the abbreviated GnuPG signatures listed in GPG_REMOTE_ID. From the test suite, an example fingerprint is:
    GPG_FINGERPRINT_ID     00CC95F05BC146B6AC4038C9E36F443C6A3FAD56
    
  • [server] When validating access.conf stanzas make sure that one of GPG_REMOTE_ID or GPG_FINGERPRINT_ID is specified whenever GnuPG signatures are to be verified for incoming SPA packets. Signature verification is the default, and can only be disabled with GPG_DISABLE_SIG but this is NOT recommended.
  • [server] Bug fix for PF firewalls without ALTQ support on FreeBSD. With this fix it doesn't matter whether ALTQ support is available or not. Thanks to Barry Allard for discovering and reporting this issue. Closes issue #121 on github.
  • [server] Bug fix discovered with the libfiu fault injection tag "fko_get_username_init" combined with valgrind analysis. This bug is only triggered after a valid authenticated and decrypted SPA packet is sniffed by fwknopd:
    ==11181== Conditional jump or move depends on uninitialised value(s)
    ==11181==    at 0x113B6D: incoming_spa (incoming_spa.c:707)
    ==11181==    by 0x11559F: process_packet (process_packet.c:211)
    ==11181==    by 0x5270857: ??? (in /usr/lib/x86_64-linux-gnu/libpcap.so.1.4.0)
    ==11181==    by 0x114BCC: pcap_capture (pcap_capture.c:270)
    ==11181==    by 0x10F32C: main (fwknopd.c:195)
    ==11181==  Uninitialised value was created by a stack allocation
    ==11181==    at 0x113476: incoming_spa (incoming_spa.c:294)
    
  • [server] Bug fix to handle SPA packets over HTTP by making sure to honor the ENABLE_SPA_OVER_HTTP fwknopd.conf variable and to properly account for SPA packet lengths when delivered via HTTP.
  • [server] Add --test mode to instruct fwknopd to acquire and process SPA packets, but not manipulate firewall rules or execute commands that are provided by SPA clients. This option is mostly useful for the fuzzing tests in the test suite to ensure broad code coverage under adverse conditions.

Software Release: fwknop-2.6.0

fwknop-2.6.0 software release The 2.6.0 release of fwknop is available for download. This release incorporates a number of feature enhancements such as an AppArmor policy for fwknopd, HMAC authenticated encryption support for the Android client, new NAT criteria that are independently configurable for each access.conf stanza, and more rigorous valgrind verification powered by the CPAN Test::Valgrind module. A few bugs were fixed as well, and similarly to the 2.5 and 2.5.1 releases, the fwknop project has a Coverity defect count of zero. As proof of this, you can see the Coverity high-level defect stats for fwknop here (you'll need to sign up for an account): Coverity Scan Build Status I would encourage any open source project that is using Coverity to publish their scan results. At last count, it appears that over 1,100 projects are using Coverity, but OpenSSH is still not one of them.

Development on fwknop-2.6.1 will begin shortly, and here is the complete ChangeLog for fwknop-2.6.0:

  • (Radostan Riedel) Added an AppArmor policy for fwknopd that is known to work on Debian and Ubuntu systems. The policy file is available at extras/apparmor/usr.sbin/fwknopd.
  • [libfko] Nikolay Kolev reported a build issue with Mac OS X Mavericks where local fwknop copies of strlcat() and strlcpy() were conflicting with those that already ship with OS X 10.9. Closes #108 on github.
  • [libfko] (Franck Joncourt) Consolidated FKO context dumping function into lib/fko_util.c. In addition to adding a shared utility function for printing an FKO context, this change also makes the FKO context output slightly easier to parse by printing each FKO attribute on a single line (this change affected the printing of the final SPA packet data). The test suite has been updated to account for this change as well.
  • [libfko] Bug fix to not attempt SPA packet decryption with GnuPG without an fko object with encryption_mode set to FKO_ENC_MODE_ASYMMETRIC. This bug was caught with valgrind validation against the perl FKO extension together with the set of SPA fuzzing packets in test/fuzzing/fuzzing_spa_packets. Note that this bug cannot be triggered via fwknopd because additional checks are made within fwknopd itself to force FKO_ENC_MODE_ASYMMETRIC whenever an access.conf stanza contains GPG key information. This fix strengthens libfko itself to independently require that the usage of fko objects without GPG key information does not result in attempted GPG decryption operations. Hence this fix applies mostly to third party usage of libfko - i.e. stock installations of fwknopd are not affected. As always, it is recommended to use HMAC authenticated encryption whenever possible even for GPG modes since this also provides a work around even for libfko prior to this fix.
  • [Android] (Gerry Reno) Updated the Android client to be compatible with Android-4.4.
  • [Android] Added HMAC support (currently optional).
  • [server] Updated pcap_dispatch() default packet count from zero to 100. This change was made to ensure backwards compatibility with older versions of libpcap per the pcap_dispatch() man page, and also because some of a report from Les Aker of an unexpected crash on Arch Linux with libpcap-1.5.1 that is fixed by this change (closes #110).
  • [server] Bug fix for SPA NAT modes on iptables firewalls to ensure that custom fwknop chains are re-created if they get deleted out from under the running fwknopd instance.
  • [server] Added FORCE_SNAT to the access.conf file so that per-access stanza SNAT criteria can be specified for SPA access.
  • [test suite] added --gdb-test to allow a previously executed fwknop or fwknopd command to be sent through gdb with the same command line args as the test suite used. This is for convenience to rapidly allow gdb to be launched when investigating fwknop/fwknopd problems.
  • [client] (Franck Joncourt) Added --stanza-list argument to show the stanza names from ~/.fwknoprc.
  • [libfko] (Hank Leininger) Contributed a patch to greatly extend libfko error code descriptions at various places in order to give much better information on what certain error conditions mean. Closes #98.
  • [test suite] Added the ability to run perl FKO module built-in tests in the t/ directory underneath the CPAN Test::Valgrind module. This allows valgrind memory checks to be applied to libfko functions via the perl FKO module (and hence rapid prototyping can be combined with memory leak detection). A check is made to see whether the Test::Valgrind module has been installed, and --enable-valgrind is also required (or --enable-all) on the test-fwknop.pl command line.

Port Knocking: Why You Should Give It Another Look

Port Knocking: Why you Should Give It Another Look (Update 10/20/2013: There is a Reddit comment thread going on this post here.)

It has been a decade since Port Knocking was first introduced to the security community in 2003, so it seemed fitting to recap how far the concept has evolved. Much effort has gone into solving architectural problems with PK systems, and today Single Packet Authorization (SPA) embodies the primary benefits of PK while fixing its limitations.

There are noted security researchers on both sides of the debate as to whether PK/SPA has any security value, but it is interesting that researchers who don't find value seem to concentrate on aspects of PK/SPA that have little to do with the chief benefit: cryptographically strong concealment. At least, this is the property offered by Single Packet Authorization but admittedly not necessarily with Port Knocking. Let's first go through some of the more common criticisms of PK/SPA, and show what the SPA answer is to each one. For those that haven't considered SPA in the past, perhaps it is time to give it a second look if for no other reason than to propose a method for breaking it.

1) Isn't PK/SPA just another password?

Suppose I hand you an arbitrary IP, say 2.2.2.2 that is running a default-drop firewall policy. As an attacker, you scan 2.2.2.2 and can't get any information back whatsoever. It doesn't respond to pings, Nmap cannot detect any TCP or UDP service under all scanning techniques, and any Metasploit module that relies on a TCP connect() call is ineffective. In the absence of a routing issue, it is safe to assume there is a firewall or ACL blocking all incoming scans. It is not feasible to tell whether there are any services listening (SSH or otherwise), and it also not feasible to tell whether there is a PK/SPA daemon either - the firewall is being used to do what it does best: block network traffic. The point is that there is no information coming back from the target. A PK/SPA daemon may be deployed, but the passive nature of PK/SPA makes it undiscoverable [1].

As a thought experiment, what would it take to make PK/SPA "just another password"? Well, if the PK/SPA daemon listened on a TCP socket and advertised itself via a server banner (like "fwknop-2.5.1, enter password for SSH access:") this would go a long way. Then Nmap would once again become an effective tool for finding the PK/SPA daemon, and an attacker could start to try different passwords. In other words, the daemon would no longer be passive, which is the whole point of PK/SPA to begin with.

But wait, you might say "but attackers can just try to brute force passive PK/SPA daemons anyway (even though they can't scan for them directly) and see if a port opens up", which brings us to:

2) Can PK/SPA be brute-forced?

Port Knocking implementations that use simple shared sequences are certainly vulnerable to brute forcing, and they are also vulnerable to replay attacks. These vulnerabilities (among other problems) were primary motivators for the development of SPA, and any modern SPA implementation is not vulnerable to either of these attacks. For example, fwknop uses AES in CBC mode authenticated with an HMAC SHA-256 in the encrypt-then-authenticate model, and both the encryption and HMAC keys (256 and 512 bits respectively for a total of 768 bits) are generated from random data in --key-gen mode. Further, fwknop can leverage GnuPG instead of AES, and 2048-bit GnuPG keys are fully supported. If it were practical to brute force fwknop encryption and authentication, then it would also be practical to brute force a lot of other cryptographic software too. Hence, fwknop is not vulnerable to such attacks in any practical sense [2].

Beyond this, from 1) above remember that the very existence of the SPA daemon is not discoverable by an attacker. For the average adversary, interacting with the SPA daemon must be done blindly "by chance". So, the target - even if it is running an SPA daemon which the attacker can't see - which implementation should the attacker try to brute force? If the target is running Moxie Marlinspike's knockknock (which uses AES in CTR mode authenticated with a truncated HMAC SHA-1), then the attacker needs to try and brute force the daemon with crafted TCP SYN packets via the following fields: TCP window size, TCP sequence, TCP acknowledgement, and the network layer IP ID. On the other hand, if the target is running fwknop, then the attacker would have to try and brute force the fwknopd daemon with UDP payloads to a port that the fwknopd pcap filter statement allows (although fwknopd can also be configured to only accept SPA payloads over ICMP instead). Should the attacker try to brute force the fwknop AES-CBC + HMAC SHA-256 mode? Or the GnuPG + HMAC SHA-256 mode? Further, the fwknopd daemon can place restrictions on the services that an authenticated client is authorized to request via the access.conf file. There are a lot of bits adding up, and the entire time the attacker doesn't even know whether an SPA daemon is actually running let alone which one.

Unfortunately for the attacker it gets worse. Even if the attacker could somehow brute force both the encryption and authentication steps in fwknop or other SPA software, to which service should the attacker try to make a connection? No service has to listen on the default port, so if a connection to SSH isn't answered should the attacker scan the target looking for a service? Maybe SPA is being used to conceal an IMAP daemon, a webserver, or OpenVPN instead of SSHD? Further, because the SPA daemon never acknowledges anything to begin with, the attacker can only infer that a brute force attempt was successful by seeing if a service is finally available after each attempt. So, in order for the attacker to be effective, the work flow for every brute force attempt should be: (1) brute force attempt, (2) scan for SSH (just because SPA is usually used to conceal SSH), (3) full scan if the SSH scan doesn't work. This starts to become extremely noisy to say the least. Even if full scans aren't also used, the volume of traffic just to attempt brute force operations by themselves is prohibitively huge.

Regardless of the attacker work flow, which service is concealed, or how heavily the attacker scans the target after every attempt, the brute force resistance offered by fwknop is fundamentally provided through the strength of cryptography. Getting past the authentication step alone would require breaking the 512-bit HMAC SHA-256 key (or forcing a hash collision against SHA-256), and fwknop even supports HMAC SHA-512 if one prefers that instead. Beyond this, the encryption key would also need to be brute-forced. For all intents and purposes it is not practical to brute force fwknop, and similar arguments apply to other SPA software.

3) Does PK/SPA add intolerable complexity?

Every security measure has some associated complexity. Firewalls add complexity. Encryption adds complexity. SSH itself adds complexity. If complexity were always the trump card against higher levels of security, then people would connect admin shells to sockets directly via netcat (or just run telnet) and not worry about encryption. People would not run firewalls because vanilla IP stacks without firewalling hooks would be simpler. Filesystem permissions overlays would be considered insecure because they add complexity. Obviously such viewpoints don't pass muster in the real world. The point is that using more complex code sometimes enables higher levels of security against widely understood threat models. For example, people use SSH and SSL because they want authentication and confidentiality over untrusted networks. Firewalls are used to reduce the attack surface that an adversary can easily communicate with. Application and/or filesystem layer policy controls are engineered to place restrictions on classes of users. The list continues.

This is not to say that complexity is not an important consideration - far from it. Rather, a real security benefit must be realized in order to justify increasing the complexity of a system. In the SPA community, we assert there is a security benefit afforded by passive, cryptographically strong service concealment. How would an attacker try to brute force user passwords via SSH when it is concealed by SPA? How would an attacker exploit even a zero-day vulnerability in a service protected by SPA? How would an attacker exploit a vulnerability in the SPA daemon itself when it is indistinguishable from a system that is running a default-drop firewall policy? (An attacker may interact with the SPA daemon, but it is more or less "by chance" [3].)

In the context of PK/SPA, the real issue is whether the complexity of the code that an attacker can interact with is more or less when PK/SPA is deployed. Anyone protecting networked services is probably already running a firewall, so the firewall usage by the PK/SPA software isn't adding to complexity that wasn't already there. Next, if PK/SPA is used to conceal multiple services (say, SSH, an IMAP daemon, and a webserver all at the same time), a would-be attacker cannot interact with any of those code bases without first getting past the PK/SPA daemon. It is a good bet that the complexity of the PK/SPA daemon is a lot less than the aggregate complexity of all three of those services if they were open to the world. Further, this may still be true even for a single daemon such as SSH as well. In essence, the effective complexity of code that an attacker can interact with may actually go down with PK/SPA deployed - that is, until the PK/SPA daemon can be circumvented (and hence a method for doing this becomes an important question for an attacker).

4) What happens if the PK/SPA daemon dies?

This is a solved problem. Process monitoring software has been around for decades. Many options exist for any OS on which a PK/SPA daemon is deployed. For example, on Ubuntu systems fwknopd is monitored by upstart. Having said this, fwknopd is extremely stable anyway so this feature is hardly ever needed in practice. Still, it is certainly important to ensure that PK/SPA usage does not cause a single point of failure, so using process monitoring software is a good idea.

Summary

There are other criticisms of SPA that are not included in this blog post, and certainly some of them are legitimate such as the fact that SPA requires a specialized client to access concealed services and the fact that "NAT piggybacking" is possible for users on the same network from which an SPA client is used when behind a NAT. However, these points don't generally rise to the level that they invalidate the SPA strategy. This blog post attempts to address those criticisms that could rise to this level were it not for the effort that has gone into solid SPA design by fwknop and other projects. More information on the design goals that guide fwknop can be found in the project tutorial.

In conclusion, SSL uses cryptography to provide authentication and confidentiality, Tor uses cryptography to provide anonymity, and SPA uses cryptography to conceal service existence. For those that assert there is no security value in the later strategy, it should consequently not be difficult to circumvent. To those in this camp, given the material in this post, please propose a method for breaking SPA.

[1] At least, a PK/SPA daemon is not discoverable by attackers who aren't already in a privileged position to sniff all traffic to and from the target. Clearly, most attackers - including password-guessing botnets - do not fall into this category.

[2] It is possible to weaken the security of fwknop SPA communications by not using --key-gen mode to generate random encryption and HMAC keys and thereby make them more susceptible to brute force attacks. However, this type of problems similarly affects other cryptographic software so it isn't unique to fwknop. And, even if a user doesn't use --key-gen mode, it is still not as easy to brute force fwknopd (which never confirms its existence to an attacker) as other software which an attacker can readily see is available to exploit.

[3] The security of fwknopd code itself is nonetheless quite important, and this is why the fwknop project uses static analysis provided by Coverity (and has a Coverity scan score of zero), the CLANG static analyzer, and also implements dynamic analysis with valgrind via a comprehensive test suite.

Software Release: fwknop-2.5 with HMAC Support

fwknop-2.5 with HMAC support After a long development cycle started over a year ago that has focused on how fwknop uses cryptography, the 2.5 release of fwknop is available for download. This release now includes support for HMAC authenticated encryption, with SHA-256 being the default digest algorithm though others such as SHA-512 are supported as well. The HMAC mode can be applied to SPA packets that have been encrypted with either Rijndael or GnuPG, and the order of operation is always encrypt-then-authenticate which is considered to be the most secure option among all possible orders. Not only does using the new HMAC mode provide a cryptographically strong authentication step for SPA communications, it also affords a significant security benefit because maliciously constructed SPA packets can be discarded before they are even sent through decryption routines. I.e. HMAC verification is a much more simplistic operation than decryption, and therefore generally less prone to programming bugs and potential security vulnerabilities.

There are many other enhancements in fwknop-2.5 as well such as usage of the Coverity static analyzer, a new ~/.fwknoprc stanza saving feature for fwknop client usage simplification, support for automatic Rijndael+HMAC key generation with the --key-gen option, many test suite improvements, an updated tutorial, and more. There is a robust roadmap for fwknop, and new releases will come faster now that a solid foundation is made upon HMAC authenticated encryption for SPA packets.

I wish to thank all who contributed to this effort - particularly Damien Stuart, Franck Joncourt, Blair Zajac, Michael T. Dean, and Ryman. Additional contributors are listed in the git history.

NOTE: If you are upgrading from a previous version of fwknop you will want to read the following information on backwards compatibility. In short, fwknop-2.5 is compatible with prior versions, but it requires one configuration tweak to either the client (add "-M legacy" to the command line) or server config (add "ENCRYPTION_MODE legacy" to each stanza in the access.conf file) if you wish to run a mixed environment of older clients and/or older servers. The reason for the incompatibility is that prior to 2.5, fwknop was not properly using PBKDF1 for Rijndael key derivation - this has been fixed.

Here is the complete ChangeLog for fwknop-2.5:

  • ***** IMPORTANT *****: If you are upgrading from an older version of fwknop, you will want to read the "Backwards Compatibility" section of the fwknop tutorial available here:

    http://www.cipherdyne.org/fwknop/docs/fwknop-tutorial.html#backwards-compatibility

    In summary, it is possible to have a mixed environment of fwknop-2.5 clients and/or servers with older client and/or servers, but this requires some configuration in order to work properly. On the server side, the directive "ENCRYPTION_MODE legacy" will need to be added to every access.conf stanza that uses Rijndael and that needs to support SPA packets from pre-2.5 clients. On the client side when generating Rijndael-encrypted SPA packets from a pre-2.5 server, the command line argument "-M legacy" will need to be given. GnuPG operations are not affected however and don't require the above steps whenever the new HMAC authenticated encryption feature (offered in fwknop-2.5) is not used.
  • Major release of new functionality - HMAC authenticated encryption support in the encrypt-then-authenticate model for SPA communications. Supported HMAC digests include MD5, SHA1, SHA256, SHA384, and SHA512. The default is HMAC-SHA256 when an HMAC is used. The HMAC mode is supported for both Rijndael and GPG encrypted SPA packet data, and provides a significant security benefit for the fwknopd server since the HMAC verification is more simplistic than decryption operations. This is particularly true for GPG. Beyond this, HMAC authenticated encryption in the encrypt-then-authenticate mode does not suffer from things like CBC-mode padding oracle attacks (see the Vaudenay attack and the more recent "Lucky 13" attack against SSL). HMAC verifications are performed with a constant time comparison function.
  • [libfko] Significant bug fix to honor the full encryption key length for user-supplied Rijndael keys > 16 bytes long. Previous to this fix, only the first 16 bytes of a key were actually used in the encryption/ decryption process even if the supplied key was longer. The result was a weakening of expected security for users that had keys > 16 bytes. Note that "passphrase" is perhaps technically a better word for "user-supplied key" in this context since the actual key is generated with the PBKDF1 key derivation algorithm. This issue was reported by Michael T. Dean. Closes issue #18 on github.
  • [libfko] Added the ability to maintain backwards compatibility with the now deprecated "zero padding" key derivation strategy in AES mode that was a hold over from the old perl fwknop implementation. This is NOT compliant with PBKDF1 and is only brought forward into fwknop-2.5 for backwards compatibility. Future versions of fwknop will remove this code altogether since PBKDF1 is now implemented.
  • [libfko+server] Ensure that all HMAC, digest, and other comparisons are done via a dedicated constant_runtime_cmp() function so that a potential attacker cannot gain any information about fail/success just by mounting a timing attack. This function always compares two buffers from beginning to end regardless of whether a difference is detected early on in the comparison, and this strategy mirrors changes in SSL libraries such as yaSSL to protect against potential timing attacks. This change fixes #85 on github which was reported by Ryman.
  • [test suite] Added --enable-openssl-checks to send all SPA packets encrypted via libfko through the OpenSSL library to ensure that the libfko usage of AES is always compatible with OpenSSL. This ensures that the fwknop usage of AES is properly implemented as verified by the OpenSSL library, which is a frequently audited high profile crypto engine. If a vulnerability is discovered in OpenSSL and a change is made, then the --enable-openssl-checks mode will allow the test suite to discover this in a automated fashion for fwknop.
  • The fwknop project is using Coverity for source code validation (in addition to other tools such as the CLANG static analyzer). Many bugs have been fixed in this release that were discovered by Coverity. These bugs spanned multiple classes of problems from memory leaks, improper use of sizeof(), potential double-free conditions, and more. Full details on these fixes are available in the git history. Any open source project that is written in a language supported by Coverity would benefit highly from participating. As of the 2.5 release, fwknop has a Coverity defect score of zero.
  • [test suite] Changed how the test suite interacts with the fwknop client and server by looking for indications that SPA packets are actually received. This is done by first waiting for 'main event loop' in fwknopd log output to ensure that fwknopd is ready to receive packets, sending the SPA packet(s), and then watching for for 'SPA Packet from IP' in fwknopd output. This is an improvement over the previous strategy that was only based on timeout values since it works identically regardless of whether fwknop is being run under valgrind or when the test suite is run on an embedded system with very limited resources. Another check is run for fwknopd receiving the SIGTERM signal to shutdown via 'fwknopd -K', and that failing, the test suite manually kills the process (though this should be rarely needed). This change was implemented based on discussions with George Herlin.
  • (Franck Joncourt) Added support for resolving hostnames in various NAT modes (fixes issue #43 in github).
  • (Franck Joncourt) Bug fix in the client for resolving hostnames in '-P icmp' mode (fixes issue #64).
  • (Franck Joncourt) Added support for saving fwknop client command line arguments via a new options --save-rc-stanza.
  • (Franck Joncourt) Added log module support for the client.
  • [client] Bug fix for --nat-rand-port mode to ensure that the port to be NAT'd is properly defined so that the fwknopd server will NAT connections to this port instead of applying the NAT operation to the port that is to be accessed via -A. This change also prints the randomly assigned port to stdout regardless of whether --verbose mode is used (since if not then the user will have no idea which port is actually going to be NAT'd on the fwknopd side).
  • (Vlad Glagolev) Submitted an OpenBSD port for fwknop-2.0.4, and this has been checked in under the extras/openbsd/fwknop-2.0.4 directory.
  • (Shawn Wilson) Added better SPA source IP logging for various fwknopd logging messages. This helps to make it more clear why certain SPA packets are rejected from some systems.
  • [client] Added --get-hmac-key to allow HMAC keys to be acquired from the specified file similarly to the --get-key option. This is a convenience only, and the fwknop rc file feature should be used instead since it is far more powerful.

ShmooCon 2013 Talk on fwknop

ShmooCon 2013 Update 03/24/2013: All presentation videos have been uploaded to shmoocon.org, and here is the full video for the talk discussed below: video.

This past weekend at ShmooCon 2013 I gave a talk entitled "Generalized Single Packet Authorization for Cloud Computing Environments" (slides, video demo), and in this blog post I thought it appropriate to give an introduction to the talk in blog format. The main point of the talk was to show that it is completely feasible to integrate SPA as implemented by fwknop with Cloud Computing environments that conform to the Infrastructure as a Service (IaaS) model. In particular, it was shown that SPA can be used to protect and access both SSH and RDP services running on EC2 instances within a Virtual Private Cloud (VPC) in Amazon's AWS environment. Accessing SSHD via SPA by itself is nothing new and has been a primary use case supported by fwknop for a long time, but maintaining a default-drop packet filtering stance for RDP is a different story due to the fact that fwknop does not support a Windows firewall. However, through the use of both SNAT and DNAT capabilities within iptables, the fwknop daemon is able to convert an Ubuntu EC2 instance running within Amazon's Cloud into an "SPA gateway" that is able to then allow access to any service running on any other system within the VPC network - including RDP on a separate Windows host. Further, such access is granted through a single externally routable Amazon Elastic IP that is associated with the Ubuntu system. The Windows host doesn't even need to have a default route back out to the Internet - it always just sees incoming RDP connections from the Ubuntu host even though they really originate out on the Internet from whatever system is allowed via the SPA protocol. Here is a graphic to illustrate this scenario: fwknop + Amazon integration If you want a video demonstration of this all in action, this demo was given during the ShmooCon talk:


Now, for more explanation, the following systems are depicted in the graphic above:

  • The SPA client system that runs the fwknop client to send SPA packets and from which SSH and RDP connections originate.
  • The Amazon border firewall.
  • An Ubuntu host on the VPC network where the fwknop daemon runs.
  • A Windows system on the VPC network where RDP lives.
At the same time, the Ubuntu VPC host is also running a webserver. This is to illustrate that everyone out on the Internet can scan the externally routable Elastic IP (107.23.221.41 in this case) associated with the Ubuntu host and see that Apache is listening on port 80. (Note that this IP is not currently associated with my Amazon account.) But, no other services are visible to a scanner on the Ubuntu system because iptables is deployed in a default-drop stance for everything except for port 80. Even so, fwknop is able to create access through the iptables policy for SSH and RDP through the Ubuntu system over port 80 (i.e. SPA "ghost services"). These two scripts that are shown in the demo video illustrate gaining access to SSH and RDP:
$ cat ssh_access.sh
#!/bin/sh -x

fwknop -A tcp/80 --server-port 40001 -R -D 107.23.221.41
sleep 2
ssh -p 80 -i ~/.ssh/cdyne-vpc-testing.pem ubuntu@107.23.221.41
exit

$ cat rdp_access.sh
#!/bin/sh -x

fwknop -A tcp/80 --server-port 40001 -N 10.0.0.223,3389 -R -D 107.23.221.41
sleep 2
rdesktop -u Administrator 107.23.221.41:80
exit

To see all of the above in action just watch the video. Hopefully the talk itself will be posted online soon, and when it is I'll post a link. More information can also be found in the talk slides.

Finally, I'd like to thank the ShmooCon organizers for putting on a great conference as usual, and I'm looking forward to next year.

Software Release - fwknop-2.0.4

fwknop-2.0.4 released The 2.0.4 release of fwknop is available for download. This release includes new support for fuzzing the SPA protocol powered by the perl FKO module along with a set of patches designed to break libfko in subtle ways, and also includes several bug fixes and small enhancements. Among the most important of these include a new check for fwknop chains on iptables firewalls after the initial fwknopd start up (so the fwknop chains can be deleted out from under the fwknop daemon without affecting SPA operations), and a new 'upstart' config for systems running the upstart daemon. Many other changes were contributed by the open source community, and these are acknowledged in the complete fwknop-2.0.4 ChangeLog below:

  • [client] Misc fixes and the addition of save_args and last command (.fwknop.last) support on the Windows platform.
  • [client] Fixed bug in username determination code where a valid value could be overrwritten in certain circumstances.
  • [server] Added upstart config at extras/upstart/fwknop.conf. This allows the fwknopd to easily be managed with upstart via commands like "service fwknop start" and "service fwknop stop".
  • [server] (Vlad Glagolev) Submitted a patch to fix ndbm/gdbm usage when --disable-file-cache is used for the autoconf configure script. This functionality was broken in be4193d734850fe60f14a26b547525ea0b9ce1e9 through improper handling of #define macros from --disable-file-cache.
  • [server] (Vlad Glagolev) Submitted a patch to fix command exec mode under SPA message type validity test. Support for command exec mode was also added to the test suite.
  • (Vlad Glagolev) Submitted an OpenBSD port for fwknop-2.0.3, and this has been checked in under the extras/openbsd/ directory.
  • [server] Bug fix to allow GPG_ALLOW_NO_PW to result in not also having to specify a Rijndael key.
  • [Android] Added new .properties files to allow the fwknop client to build and function properly on the latest Android release (4.1.2).
  • [client] Added '-P udpraw' to allow the client to send SPA packets over UDP with a spoofed source IP address. This is in addition to the original 'tcpraw' and 'icmp' protocols that also support a spoofed source IP.
  • [libfko] Bug fix to check b64_decode() return value to ensure that non-base64 encoded data is never used. Even though other validation routines checked decoded results, it is important to discard invalid data as early as possible. Note too that such invalid data would only be provided to b64_decode() after proper decryption, so the client must provide authentic SPA data.
  • [libfko] Added validation of NAT access strings in the various NAT modes.
  • [libfko] Restricted usernames embedded in SPA packets to be alpha-numeric along with "-" chars.
  • [client] (Franck Joncourt) Contributed a patch to allow the fwknop client to be stopped during the password entry prompt with Ctrl-C before any SPA packet is sent on the wire.
  • [client+server] Applied patch from Franck Joncourt to remove unnecessary chmod() call when creating client rc file and server replay cache file. The permissions are now set appropriately via open(), and at the same time this patch fixes a potential race condition since the previous code used fopen() followed by chmod().
  • [server] Bug fix to accept SPA packets over ICMP if the fwknop client is executed with '-P icmp' and the user has the required privileges.
  • [test suite] Applied patch from Franck Joncourt to have the perl FKO module link against libfko in the local directory (if it exists) so that it doesn't have to have libfko completely installed in the /usr/lib/ directory. This allows the test suite to run FKO tests without installing libfko.
  • [test suite] Significant update to include a set of fuzzing SPA packets that are built using a patched version of libfko. These packets are located in the test/fuzzing/bogus_spa_packets file, and are designed to ensure proper validation of SPA packet data. This validation is performed in --enable-perl-module-checks mode via the perl FKO module.
  • [client] Added --icmp-type and --icmp-code arguments so the user can control the icmp type/code combination for spoofed SPA packets ('-P icmp') mode.
  • [client] Updated default TTL value to 64 for spoofed SPA packets. This is closer to more OS default TTL values than the previous 255.
  • Updated build CFLAGS and LDFLAGS to conform to the Debian hardening-includes file for PIE support (e.g. '-fPIE' for CFLAGS and '-fPIE -pie' for LDFLAGS).
  • [test suite] For GnuPG tests that require a passphrase associated with a gpg key, added a pinentry check to see if the local gpg engine requires it. If so, the gpg test that require a key are excluded since.
  • [server] Added a new '--pcap-file <file>' option to allow pcap files to be processed directly by fwknopd instead of sniffing an interface. This feature is mostly intended for debugging purposes.
  • [server] Added chain_exists() check to SPA rule creation so that if any of the fwknop chains are deleted out from under fwknopd they will be recreated on the fly. This mitigates scenarios where fwknopd might be started before a system level firewall policy is applied due to init script ordering, or if an iptables policy is re-applied without restarting fwknopd.
The complete fwknop-2.0.4 ChangeLog can also be found here via the fwknop gitweb interface.

Tutorial on Single Packet Authorization with fwknop

Single Packet Authorization: The fwknop Approach One thing that has been requested a few times is a tutorial on how to use fwknop, so here it is: "Single Packet Authorization: A Comprehensive Guide to Strong Service Hardening with fwknop". This document attempts to cover most usages of fwknop to wrap an SPA layer around arbitrary services, though some less used scenarios such as SPA over Tor and SPA packet spoofing are covered as well. Please contact me if you find any errors, or if you would like to see material added on a particular fwknop use case. The complete table of contents appears below:

Single Packet Authorization: The fwknop Approach

Single Packet Authorization: The fwknop Approach Update: The fwknop design material in this blog post has been updated in the fwknop tutorial.

There are many different port knocking implementations out there, and a few that take things to the next level with Single Packet Authorization. All PK/SPA implementations have three primary goals: 1) the deployment of a default-drop firewall policy for a service (such as SSHD) in order to protect against scans, potential vulnerabilities, and brute force password guessing attempts, 2) the passive monitoring of specially constructed authentication information from the PK/SPA client, and 3) the dynamic reconfiguration of the firewall to allow temporary access to the concealed service.

The PK/SPA server never acknowledges any data from the client, so it is infeasible to scan not only for any properly implemented PK/SPA server, but also for any protected service. Effectively, services become invisible to the Internet - except for those that can produce the necessary authentication information, or those that are in a privileged position to sniff traffic between an authenticated PK/SPA client and the target service.

Users of firewalls find value in the idea that traffic to a service can be blocked from all but a few pre-defined networks according to the firewall policy. Few people question whether this is valuable from a security perspective - firewalls generally enhance security (the occasional firewall vulnerability not withstanding). The PK/SPA strategy extends the notion of filtering traffic for a set of services by adding a lightweight crypto layer to allow temporary access from networks that cannot be anticipated when the firewall policy is written. This provides concealment by default, and the SPA strategy asserts that there is value in this. This can apply to all sorts of services from SSHD and OpenVPN to mail protocols like POP and IMAP and even to HTTP. Anyone scanning for services to target will never see a service protected by SPA.

If the above describes unifying aspects of PK/SPA software, this is where the similarities in PK/SPA software ends.

Below are the design choices that guide fwknop development. Other PK/SPA software make different design choices, and whether you prefer fwknop vs. another implementation depends at least partially on whether you agree with the following:

  • No heavyweight interpreted languages:
    Many people don't want to install perl, python, or ruby on a firewall or network gateway device. Such languages are large and complex, and are implemented by similarly complex binaries that are usually themselves written in a language like C. Firewalls are frequently stripped down systems that are designed to just filter network traffic, provide administrative interfaces and sometimes VPN services, and not do much else. Both the fwknop client and server are entirely written in C - there is no requirement for perl, python, or any other interpreted language.

  • Support embedded devices:
    A consequence of fwknop being developed in C is that it only uses minimal system resources and can therefore support embedded devices that don't have a lot of computing power or main memory. For example, where the older perl version of fwknop could not run on Linksys routers supported by OpenWRT, these same routers can run the newer C version of fwknop.

  • Don't require admin access to run the SPA client:
    There are many computing environments where users don't have privileged accounts. This should not present a barrier to using the SPA client. In fwknop, SPA packets are (by default) sent over a regular UDP socket and therefore require no special privileges.

  • Don't require the manipulation of raw packet headers:
    There are some SPA implementations that communicate information within specially modified fields within IP or TCP headers. Such manipulation requires that the user have the ability to acquire a raw socket from the OS, and this requires admin level privileges. In addition, depending on how the SPA client manipulates packet headers when building an SPA packet, other monitoring systems such as an IDS or a passive OS fingerprinter may produce event data that unnecessarily calls attention to the SPA communications. This is not to say that that is impossible to detect SPA packets generated by fwknop - it is just that a monitoring system is more likely to flag communications that involve manipulated packet headers than to generate an event for packets produced by fwknop. For example, intrusion detection systems track TCP connections, and spoofed TCP ACK's that are not part of a proper connection (assuming non-asymmetric routing) may potentially be flagged. Also, manipulated TCP options fields that don't conform to OS defaults will cause an OS to appear to change under the observation of things like p0f. While sometimes this is an expected behavior such as if a VM is being run or a system is actually a NAT device with other systems behind it, there are plenty of deployment scenarios where this is not expected. Addressing this issue at length could be a separate blog post in itself.

  • Don't trust the IP header:
    Any SPA implementation that trusts the source IP address in the IP header of the SPA packet is vulnerable to a MITM attack. An inline device that can intercept an SPA packet can hold the original packet and retransmit it but with the source IP changed to whatever the attacker wants. If the SPA server relies on the source IP in the IP header, then it has no way to know that it isn't legitimate. If an SPA implementation is going to go to the trouble of leveraging cryptography - certainly important since replay attacks among other problems can't be prevented without it - then the IP should be made part of an encrypted payload. This is exactly what fwknop does with the -a or -R arguments on the fwknop command line. An attacker can intercept an SPA packet produced by fwknop, change the source IP and retransmit, but SPA server will only allow access to the IP that was originally encrypted within the SPA payload.

  • Support server-side NAT:
    There are plenty of networks with a border firewall where a remote user actually wants access to a service that is running on an internal system and not on the firewall itself. Typical ways of accessing such a service involve running VPN software, but with an SPA implementation that can manipulate NAT rules on the border firewall it is possible to transparently grant SPA users access to internal services through the firewall. With fwknop, the capability to leverage server-side NAT has been built in for a long time.

  • Support multiple firewalls:
    Because fwknop does not rely on specialized logging infrastructure or link against libraries that are tied to one firewall architecture or another, it can easily support multiple firewalls just by executing the local firewall admin command line interface. It currently supports iptables on Linux, ipfw on FreeBSD and Mac OS X, and pf on OpenBSD.

  • Minimize library dependencies:
    Given the design decisions made by fwknop above, it is important to minimize library dependencies and to audit the source code. Here are all library dependencies in the fwknopd daemon including GnuPG support (this is optional):
    $ ldd ./server/.libs/fwknopd
        linux-vdso.so.1 =>  (0x00007ffeebf820e0)
        libfko.so.0 => /usr/lib/libfko.so.0 (0x00007f1a6ae930e0)
        libpcap.so.0.8 => /usr/lib/x86_64-linux-gnu/libpcap.so.0.8 (0x00007e1a6a85c0e0)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007e1a6a49e0a0)
        libgpgme.so.11 => /usr/lib/libgpgme.so.11 (0x00007f1aeaee800e)
        /lib64/ld-linux-x86-64.so.2 (0x00007e1a6aede0e0)
        libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x0000ef1a6a06e0e0)
    
    The libfko library is developed by the fwknop project for SPA encryption, decryption, and validation. libpcap is a standard C library that is leveraged by a lot of security projects including intrusion detection systems and more. The gpgme libraries are only used if you want GnuPG support, and the remaining libraries are standard system libraries that typical Linux binaries need. That's it. One may certainly question the use of libpcap, and in defense of this choice note that: 1) fwknopd by default does not put the sniffing interface into promiscuous mode, 2) fwknopd by default only pays attention to UDP packets to port 62201, 3) if you examine Metasploit you'll see that there only a few exploits for pcap-based software and they target what rides on top of libpcap vs. libpcap itself, and 4) even if there is a vulnerability in libpcap, the exploit model is different than it is for normal server software that you can easily scan for, and being different in this game is good.

  • Support both symmetric and asymmetric ciphers:
    Some users prefer the security properties that asymmetric crypto systems have over those provided by symmetric algorithms like Rijndael, or vice versa. fwknop supports both Rijndael and GnuPG for SPA encryption/decryption.

  • Support HMAC in the encrypt-then-authenticate model:
    Authenticated encryption with an HMAC is supported by fwknop as of the 2.5 release for both symmetric and asymmetric encryption modes. The implementation is careful to apply an HMAC to SPA packets according to the encrypt-then-authenticate model which provides strong resistance to certain kinds of cryptanalytic attacks. In addition, on the fwknopd server side, verification of the HMAC is a more simplistic operation than sending data through a set of decryption routines, so this provides an additional security benefit by reducing the complexity of code the malicious traffic can interact with. This is particularly true for SPA packets that are encrypted with GnuPG.