Michael Rash, Security Researcher

fwknop    [Summary View]

Next »

Software Release: fwknop-2.6.0

fwknop-2.6.0 software release The 2.6.0 release of fwknop is available for download. This release incorporates a number of feature enhancements such as an AppArmor policy for fwknopd, HMAC authenticated encryption support for the Android client, new NAT criteria that are independently configurable for each access.conf stanza, and more rigorous valgrind verification powered by the CPAN Test::Valgrind module. A few bugs were fixed as well, and similarly to the 2.5 and 2.5.1 releases, the fwknop project has a Coverity defect count of zero. As proof of this, you can see the Coverity high-level defect stats for fwknop here (you'll need to sign up for an account): Coverity Scan Build Status I would encourage any open source project that is using Coverity to publish their scan results. At last count, it appears that over 1,100 projects are using Coverity, but OpenSSH is still not one of them.

Development on fwknop-2.6.1 will begin shortly, and here is the complete ChangeLog for fwknop-2.6.0:

  • (Radostan Riedel) Added an AppArmor policy for fwknopd that is known to work on Debian and Ubuntu systems. The policy file is available at extras/apparmor/usr.sbin/fwknopd.
  • [libfko] Nikolay Kolev reported a build issue with Mac OS X Mavericks where local fwknop copies of strlcat() and strlcpy() were conflicting with those that already ship with OS X 10.9. Closes #108 on github.
  • [libfko] (Franck Joncourt) Consolidated FKO context dumping function into lib/fko_util.c. In addition to adding a shared utility function for printing an FKO context, this change also makes the FKO context output slightly easier to parse by printing each FKO attribute on a single line (this change affected the printing of the final SPA packet data). The test suite has been updated to account for this change as well.
  • [libfko] Bug fix to not attempt SPA packet decryption with GnuPG without an fko object with encryption_mode set to FKO_ENC_MODE_ASYMMETRIC. This bug was caught with valgrind validation against the perl FKO extension together with the set of SPA fuzzing packets in test/fuzzing/fuzzing_spa_packets. Note that this bug cannot be triggered via fwknopd because additional checks are made within fwknopd itself to force FKO_ENC_MODE_ASYMMETRIC whenever an access.conf stanza contains GPG key information. This fix strengthens libfko itself to independently require that the usage of fko objects without GPG key information does not result in attempted GPG decryption operations. Hence this fix applies mostly to third party usage of libfko - i.e. stock installations of fwknopd are not affected. As always, it is recommended to use HMAC authenticated encryption whenever possible even for GPG modes since this also provides a work around even for libfko prior to this fix.
  • [Android] (Gerry Reno) Updated the Android client to be compatible with Android-4.4.
  • [Android] Added HMAC support (currently optional).
  • [server] Updated pcap_dispatch() default packet count from zero to 100. This change was made to ensure backwards compatibility with older versions of libpcap per the pcap_dispatch() man page, and also because some of a report from Les Aker of an unexpected crash on Arch Linux with libpcap-1.5.1 that is fixed by this change (closes #110).
  • [server] Bug fix for SPA NAT modes on iptables firewalls to ensure that custom fwknop chains are re-created if they get deleted out from under the running fwknopd instance.
  • [server] Added FORCE_SNAT to the access.conf file so that per-access stanza SNAT criteria can be specified for SPA access.
  • [test suite] added --gdb-test to allow a previously executed fwknop or fwknopd command to be sent through gdb with the same command line args as the test suite used. This is for convenience to rapidly allow gdb to be launched when investigating fwknop/fwknopd problems.
  • [client] (Franck Joncourt) Added --stanza-list argument to show the stanza names from ~/.fwknoprc.
  • [libfko] (Hank Leininger) Contributed a patch to greatly extend libfko error code descriptions at various places in order to give much better information on what certain error conditions mean. Closes #98.
  • [test suite] Added the ability to run perl FKO module built-in tests in the t/ directory underneath the CPAN Test::Valgrind module. This allows valgrind memory checks to be applied to libfko functions via the perl FKO module (and hence rapid prototyping can be combined with memory leak detection). A check is made to see whether the Test::Valgrind module has been installed, and --enable-valgrind is also required (or --enable-all) on the command line.

Validating libfko Memory Usage with Test::Valgrind

Validating libfko Memory Usage with Test::Valgrind The fwknop project consistently uses valgrind to ensure that memory leaks, double free() conditions, and other problems do not creep into the code base. A high level of automation is built around valgrind usage with the fwknop test suite, and a recent addition extends this even further by using the excellent CPAN Test::Valgrind module. Even though the test suite has had the ability to run tests through valgrind, previous to this change these tests only applied to the fwknop C binaries when executed directly by the test suite. Further, some of the most rigorous testing is done through the usage of the perl FKO extension to fuzz libfko functions, so without the Test::Valgrind module these tests also could not take advantage of valgrind support. Now that the test suite supports Test::Valgrind (and a check is done to see if it is installed), all fuzzing tests can also be validated with valgrind. Technically, the fuzzing tests have been added as FKO built-in tests in the t/ directory, and the test suite runs them through Test::Valgrind like this:
# prove --exec 'perl -Iblib/lib -Iblib/arch -MTest::Valgrind' t/*.t
Here is a complete example - first, run the test suite like so:
# ./ --enable-all --include perl --test-limit 3

[+] Starting the fwknop test suite...

    args: --enable-all --include perl --test-limit 3

[+] Total test buckets to execute: 3

[perl FKO module] [compile/install] to: ./FKO...................pass (1)
[perl FKO module] [make test] run built-in tests................pass (2)
[perl FKO module] [prove t/*.t] Test::Valgrind..................pass (3)
[valgrind output] [flagged functions] ..........................pass (4)

    Run time: 1.27 minutes

[+] 0/0/0 OpenSSL tests passed/failed/executed
[+] 0/0/0 OpenSSL HMAC tests passed/failed/executed
[+] 4/0/4 test buckets passed/failed/executed
Note that all tests passed as shown above. This indicates that the test suite has not found any memory leaks through the fuzzing tests run via Test::Valgrind. But, let's validate this by artificially introducing a memory leak and see if the test suite can automatically catch it. For example, here is a patch that forces a memory leak in the validate_access_msg() libfko function. This function ensures that the shape of the access request conforms to something fwknop expects like ",tcp/22". The memory leak happens because a new buffer is allocated from the heap but is never free()'d before returning from the function (obviously this patch is for illustration and testing purposes only):
$ git diff
diff --git a/lib/fko_message.c b/lib/fko_message.c
index fa6803b..c04e035 100644
--- a/lib/fko_message.c
+++ b/lib/fko_message.c
@@ -251,6 +251,13 @@ validate_access_msg(const char *msg)
     const char   *ndx;
     int     res         = FKO_SUCCESS;
     int     startlen    = strnlen(msg, MAX_SPA_MESSAGE_SIZE);
+    char *leak = NULL;
+    leak = malloc(100);
+    leak[0] = 'a';
+    leak[1] = 'a';
+    leak[2] = '\0';
+    printf("LEAK: %s\n", leak);

     if(startlen == MAX_SPA_MESSAGE_SIZE)
Now recompile fwknop and run the test suite again, after applying the patch (recompilation output is not shown):
# cd ../
# make
# test
# ./ --enable-all --include perl --test-limit 3
[+] Starting the fwknop test suite...

    args: --enable-all --include perl --test-limit 3

    Saved results from previous run to: output.last/

    Valgrind mode enabled, will import previous coverage from:

[+] Total test buckets to execute: 3

[perl FKO module] [compile/install] to: ./FKO...................pass (1)
[perl FKO module] [make test] run built-in tests................pass (2)
[perl FKO module] [prove t/*.t] (3)
[valgrind output] [flagged functions] (4)

    Run time: 1.27 minutes

[+] 0/0/0 OpenSSL tests passed/failed/executed
[+] 0/0/0 OpenSSL HMAC tests passed/failed/executed
[+] 2/2/4 test buckets passed/failed/executed

This time two tests fail. The first is the test that runs the perl FKO module built-in tests under Test::Valgrind, and the second is the "flagged functions" test which compares test suite output looking for new functions that valgrind has flagged vs. the previous test suite execution. By looking at the output file of the "flagged functions" test it is easy to see the offending function where the new memory leak exists. This provides an easy, automated way of memory leak detection that is driven by perl FKO fuzzing tests.
# cat output/4.test
[+] fwknop client functions (with call line numbers):
       10 : validate_access_msg [fko_message.c:256]
        6 : fko_set_spa_message [fko_message.c:184]
        4 : fko_new_with_data [fko_funcs.c:263]
        4 : fko_decrypt_spa_data [fko_encryption.c:264]
        4 : fko_decode_spa_data [fko_decode.c:350]
Currently, there are no known memory leaks in the fwknop code, and automation built around the Test::Valgrind module will help keep it that way.

Port Knocking: Why You Should Give It Another Look

Port Knocking: Why you Should Give It Another Look (Update 10/20/2013: There is a Reddit comment thread going on this post here.)

It has been a decade since Port Knocking was first introduced to the security community in 2003, so it seemed fitting to recap how far the concept has evolved. Much effort has gone into solving architectural problems with PK systems, and today Single Packet Authorization (SPA) embodies the primary benefits of PK while fixing its limitations.

There are noted security researchers on both sides of the debate as to whether PK/SPA has any security value, but it is interesting that researchers who don't find value seem to concentrate on aspects of PK/SPA that have little to do with the chief benefit: cryptographically strong concealment. At least, this is the property offered by Single Packet Authorization but admittedly not necessarily with Port Knocking. Let's first go through some of the more common criticisms of PK/SPA, and show what the SPA answer is to each one. For those that haven't considered SPA in the past, perhaps it is time to give it a second look if for no other reason than to propose a method for breaking it.

1) Isn't PK/SPA just another password?

Suppose I hand you an arbitrary IP, say that is running a default-drop firewall policy. As an attacker, you scan and can't get any information back whatsoever. It doesn't respond to pings, Nmap cannot detect any TCP or UDP service under all scanning techniques, and any Metasploit module that relies on a TCP connect() call is ineffective. In the absence of a routing issue, it is safe to assume there is a firewall or ACL blocking all incoming scans. It is not feasible to tell whether there are any services listening (SSH or otherwise), and it also not feasible to tell whether there is a PK/SPA daemon either - the firewall is being used to do what it does best: block network traffic. The point is that there is no information coming back from the target. A PK/SPA daemon may be deployed, but the passive nature of PK/SPA makes it undiscoverable [1].

As a thought experiment, what would it take to make PK/SPA "just another password"? Well, if the PK/SPA daemon listened on a TCP socket and advertised itself via a server banner (like "fwknop-2.5.1, enter password for SSH access:") this would go a long way. Then Nmap would once again become an effective tool for finding the PK/SPA daemon, and an attacker could start to try different passwords. In other words, the daemon would no longer be passive, which is the whole point of PK/SPA to begin with.

But wait, you might say "but attackers can just try to brute force passive PK/SPA daemons anyway (even though they can't scan for them directly) and see if a port opens up", which brings us to:

2) Can PK/SPA be brute-forced?

Port Knocking implementations that use simple shared sequences are certainly vulnerable to brute forcing, and they are also vulnerable to replay attacks. These vulnerabilities (among other problems) were primary motivators for the development of SPA, and any modern SPA implementation is not vulnerable to either of these attacks. For example, fwknop uses AES in CBC mode authenticated with an HMAC SHA-256 in the encrypt-then-authenticate model, and both the encryption and HMAC keys (256 and 512 bits respectively for a total of 768 bits) are generated from random data in --key-gen mode. Further, fwknop can leverage GnuPG instead of AES, and 2048-bit GnuPG keys are fully supported. If it were practical to brute force fwknop encryption and authentication, then it would also be practical to brute force a lot of other cryptographic software too. Hence, fwknop is not vulnerable to such attacks in any practical sense [2].

Beyond this, from 1) above remember that the very existence of the SPA daemon is not discoverable by an attacker. For the average adversary, interacting with the SPA daemon must be done blindly "by chance". So, the target - even if it is running an SPA daemon which the attacker can't see - which implementation should the attacker try to brute force? If the target is running Moxie Marlinspike's knockknock (which uses AES in CTR mode authenticated with a truncated HMAC SHA-1), then the attacker needs to try and brute force the daemon with crafted TCP SYN packets via the following fields: TCP window size, TCP sequence, TCP acknowledgement, and the network layer IP ID. On the other hand, if the target is running fwknop, then the attacker would have to try and brute force the fwknopd daemon with UDP payloads to a port that the fwknopd pcap filter statement allows (although fwknopd can also be configured to only accept SPA payloads over ICMP instead). Should the attacker try to brute force the fwknop AES-CBC + HMAC SHA-256 mode? Or the GnuPG + HMAC SHA-256 mode? Further, the fwknopd daemon can place restrictions on the services that an authenticated client is authorized to request via the access.conf file. There are a lot of bits adding up, and the entire time the attacker doesn't even know whether an SPA daemon is actually running let alone which one.

Unfortunately for the attacker it gets worse. Even if the attacker could somehow brute force both the encryption and authentication steps in fwknop or other SPA software, to which service should the attacker try to make a connection? No service has to listen on the default port, so if a connection to SSH isn't answered should the attacker scan the target looking for a service? Maybe SPA is being used to conceal an IMAP daemon, a webserver, or OpenVPN instead of SSHD? Further, because the SPA daemon never acknowledges anything to begin with, the attacker can only infer that a brute force attempt was successful by seeing if a service is finally available after each attempt. So, in order for the attacker to be effective, the work flow for every brute force attempt should be: (1) brute force attempt, (2) scan for SSH (just because SPA is usually used to conceal SSH), (3) full scan if the SSH scan doesn't work. This starts to become extremely noisy to say the least. Even if full scans aren't also used, the volume of traffic just to attempt brute force operations by themselves is prohibitively huge.

Regardless of the attacker work flow, which service is concealed, or how heavily the attacker scans the target after every attempt, the brute force resistance offered by fwknop is fundamentally provided through the strength of cryptography. Getting past the authentication step alone would require breaking the 512-bit HMAC SHA-256 key (or forcing a hash collision against SHA-256), and fwknop even supports HMAC SHA-512 if one prefers that instead. Beyond this, the encryption key would also need to be brute-forced. For all intents and purposes it is not practical to brute force fwknop, and similar arguments apply to other SPA software.

3) Does PK/SPA add intolerable complexity?

Every security measure has some associated complexity. Firewalls add complexity. Encryption adds complexity. SSH itself adds complexity. If complexity were always the trump card against higher levels of security, then people would connect admin shells to sockets directly via netcat (or just run telnet) and not worry about encryption. People would not run firewalls because vanilla IP stacks without firewalling hooks would be simpler. Filesystem permissions overlays would be considered insecure because they add complexity. Obviously such viewpoints don't pass muster in the real world. The point is that using more complex code sometimes enables higher levels of security against widely understood threat models. For example, people use SSH and SSL because they want authentication and confidentiality over untrusted networks. Firewalls are used to reduce the attack surface that an adversary can easily communicate with. Application and/or filesystem layer policy controls are engineered to place restrictions on classes of users. The list continues.

This is not to say that complexity is not an important consideration - far from it. Rather, a real security benefit must be realized in order to justify increasing the complexity of a system. In the SPA community, we assert there is a security benefit afforded by passive, cryptographically strong service concealment. How would an attacker try to brute force user passwords via SSH when it is concealed by SPA? How would an attacker exploit even a zero-day vulnerability in a service protected by SPA? How would an attacker exploit a vulnerability in the SPA daemon itself when it is indistinguishable from a system that is running a default-drop firewall policy? (An attacker may interact with the SPA daemon, but it is more or less "by chance" [3].)

In the context of PK/SPA, the real issue is whether the complexity of the code that an attacker can interact with is more or less when PK/SPA is deployed. Anyone protecting networked services is probably already running a firewall, so the firewall usage by the PK/SPA software isn't adding to complexity that wasn't already there. Next, if PK/SPA is used to conceal multiple services (say, SSH, an IMAP daemon, and a webserver all at the same time), a would-be attacker cannot interact with any of those code bases without first getting past the PK/SPA daemon. It is a good bet that the complexity of the PK/SPA daemon is a lot less than the aggregate complexity of all three of those services if they were open to the world. Further, this may still be true even for a single daemon such as SSH as well. In essence, the effective complexity of code that an attacker can interact with may actually go down with PK/SPA deployed - that is, until the PK/SPA daemon can be circumvented (and hence a method for doing this becomes an important question for an attacker).

4) What happens if the PK/SPA daemon dies?

This is a solved problem. Process monitoring software has been around for decades. Many options exist for any OS on which a PK/SPA daemon is deployed. For example, on Ubuntu systems fwknopd is monitored by upstart. Having said this, fwknopd is extremely stable anyway so this feature is hardly ever needed in practice. Still, it is certainly important to ensure that PK/SPA usage does not cause a single point of failure, so using process monitoring software is a good idea.


There are other criticisms of SPA that are not included in this blog post, and certainly some of them are legitimate such as the fact that SPA requires a specialized client to access concealed services and the fact that "NAT piggybacking" is possible for users on the same network from which an SPA client is used when behind a NAT. However, these points don't generally rise to the level that they invalidate the SPA strategy. This blog post attempts to address those criticisms that could rise to this level were it not for the effort that has gone into solid SPA design by fwknop and other projects. More information on the design goals that guide fwknop can be found in the project tutorial.

In conclusion, SSL uses cryptography to provide authentication and confidentiality, Tor uses cryptography to provide anonymity, and SPA uses cryptography to conceal service existence. For those that assert there is no security value in the later strategy, it should consequently not be difficult to circumvent. To those in this camp, given the material in this post, please propose a method for breaking SPA.

[1] At least, a PK/SPA daemon is not discoverable by attackers who aren't already in a privileged position to sniff all traffic to and from the target. Clearly, most attackers - including password-guessing botnets - do not fall into this category.

[2] It is possible to weaken the security of fwknop SPA communications by not using --key-gen mode to generate random encryption and HMAC keys and thereby make them more susceptible to brute force attacks. However, this type of problems similarly affects other cryptographic software so it isn't unique to fwknop. And, even if a user doesn't use --key-gen mode, it is still not as easy to brute force fwknopd (which never confirms its existence to an attacker) as other software which an attacker can readily see is available to exploit.

[3] The security of fwknopd code itself is nonetheless quite important, and this is why the fwknop project uses static analysis provided by Coverity (and has a Coverity scan score of zero), the CLANG static analyzer, and also implements dynamic analysis with valgrind via a comprehensive test suite.

Software Release: fwknop-2.5 with HMAC Support

fwknop-2.5 with HMAC support After a long development cycle started over a year ago that has focused on how fwknop uses cryptography, the 2.5 release of fwknop is available for download. This release now includes support for HMAC authenticated encryption, with SHA-256 being the default digest algorithm though others such as SHA-512 are supported as well. The HMAC mode can be applied to SPA packets that have been encrypted with either Rijndael or GnuPG, and the order of operation is always encrypt-then-authenticate which is considered to be the most secure option among all possible orders. Not only does using the new HMAC mode provide a cryptographically strong authentication step for SPA communications, it also affords a significant security benefit because maliciously constructed SPA packets can be discarded before they are even sent through decryption routines. I.e. HMAC verification is a much more simplistic operation than decryption, and therefore generally less prone to programming bugs and potential security vulnerabilities.

There are many other enhancements in fwknop-2.5 as well such as usage of the Coverity static analyzer, a new ~/.fwknoprc stanza saving feature for fwknop client usage simplification, support for automatic Rijndael+HMAC key generation with the --key-gen option, many test suite improvements, an updated tutorial, and more. There is a robust roadmap for fwknop, and new releases will come faster now that a solid foundation is made upon HMAC authenticated encryption for SPA packets.

I wish to thank all who contributed to this effort - particularly Damien Stuart, Franck Joncourt, Blair Zajac, Michael T. Dean, and Ryman. Additional contributors are listed in the git history.

NOTE: If you are upgrading from a previous version of fwknop you will want to read the following information on backwards compatibility. In short, fwknop-2.5 is compatible with prior versions, but it requires one configuration tweak to either the client (add "-M legacy" to the command line) or server config (add "ENCRYPTION_MODE legacy" to each stanza in the access.conf file) if you wish to run a mixed environment of older clients and/or older servers. The reason for the incompatibility is that prior to 2.5, fwknop was not properly using PBKDF1 for Rijndael key derivation - this has been fixed.

Here is the complete ChangeLog for fwknop-2.5:

  • ***** IMPORTANT *****: If you are upgrading from an older version of fwknop, you will want to read the "Backwards Compatibility" section of the fwknop tutorial available here:

    In summary, it is possible to have a mixed environment of fwknop-2.5 clients and/or servers with older client and/or servers, but this requires some configuration in order to work properly. On the server side, the directive "ENCRYPTION_MODE legacy" will need to be added to every access.conf stanza that uses Rijndael and that needs to support SPA packets from pre-2.5 clients. On the client side when generating Rijndael-encrypted SPA packets from a pre-2.5 server, the command line argument "-M legacy" will need to be given. GnuPG operations are not affected however and don't require the above steps whenever the new HMAC authenticated encryption feature (offered in fwknop-2.5) is not used.
  • Major release of new functionality - HMAC authenticated encryption support in the encrypt-then-authenticate model for SPA communications. Supported HMAC digests include MD5, SHA1, SHA256, SHA384, and SHA512. The default is HMAC-SHA256 when an HMAC is used. The HMAC mode is supported for both Rijndael and GPG encrypted SPA packet data, and provides a significant security benefit for the fwknopd server since the HMAC verification is more simplistic than decryption operations. This is particularly true for GPG. Beyond this, HMAC authenticated encryption in the encrypt-then-authenticate mode does not suffer from things like CBC-mode padding oracle attacks (see the Vaudenay attack and the more recent "Lucky 13" attack against SSL). HMAC verifications are performed with a constant time comparison function.
  • [libfko] Significant bug fix to honor the full encryption key length for user-supplied Rijndael keys > 16 bytes long. Previous to this fix, only the first 16 bytes of a key were actually used in the encryption/ decryption process even if the supplied key was longer. The result was a weakening of expected security for users that had keys > 16 bytes. Note that "passphrase" is perhaps technically a better word for "user-supplied key" in this context since the actual key is generated with the PBKDF1 key derivation algorithm. This issue was reported by Michael T. Dean. Closes issue #18 on github.
  • [libfko] Added the ability to maintain backwards compatibility with the now deprecated "zero padding" key derivation strategy in AES mode that was a hold over from the old perl fwknop implementation. This is NOT compliant with PBKDF1 and is only brought forward into fwknop-2.5 for backwards compatibility. Future versions of fwknop will remove this code altogether since PBKDF1 is now implemented.
  • [libfko+server] Ensure that all HMAC, digest, and other comparisons are done via a dedicated constant_runtime_cmp() function so that a potential attacker cannot gain any information about fail/success just by mounting a timing attack. This function always compares two buffers from beginning to end regardless of whether a difference is detected early on in the comparison, and this strategy mirrors changes in SSL libraries such as yaSSL to protect against potential timing attacks. This change fixes #85 on github which was reported by Ryman.
  • [test suite] Added --enable-openssl-checks to send all SPA packets encrypted via libfko through the OpenSSL library to ensure that the libfko usage of AES is always compatible with OpenSSL. This ensures that the fwknop usage of AES is properly implemented as verified by the OpenSSL library, which is a frequently audited high profile crypto engine. If a vulnerability is discovered in OpenSSL and a change is made, then the --enable-openssl-checks mode will allow the test suite to discover this in a automated fashion for fwknop.
  • The fwknop project is using Coverity for source code validation (in addition to other tools such as the CLANG static analyzer). Many bugs have been fixed in this release that were discovered by Coverity. These bugs spanned multiple classes of problems from memory leaks, improper use of sizeof(), potential double-free conditions, and more. Full details on these fixes are available in the git history. Any open source project that is written in a language supported by Coverity would benefit highly from participating. As of the 2.5 release, fwknop has a Coverity defect score of zero.
  • [test suite] Changed how the test suite interacts with the fwknop client and server by looking for indications that SPA packets are actually received. This is done by first waiting for 'main event loop' in fwknopd log output to ensure that fwknopd is ready to receive packets, sending the SPA packet(s), and then watching for for 'SPA Packet from IP' in fwknopd output. This is an improvement over the previous strategy that was only based on timeout values since it works identically regardless of whether fwknop is being run under valgrind or when the test suite is run on an embedded system with very limited resources. Another check is run for fwknopd receiving the SIGTERM signal to shutdown via 'fwknopd -K', and that failing, the test suite manually kills the process (though this should be rarely needed). This change was implemented based on discussions with George Herlin.
  • (Franck Joncourt) Added support for resolving hostnames in various NAT modes (fixes issue #43 in github).
  • (Franck Joncourt) Bug fix in the client for resolving hostnames in '-P icmp' mode (fixes issue #64).
  • (Franck Joncourt) Added support for saving fwknop client command line arguments via a new options --save-rc-stanza.
  • (Franck Joncourt) Added log module support for the client.
  • [client] Bug fix for --nat-rand-port mode to ensure that the port to be NAT'd is properly defined so that the fwknopd server will NAT connections to this port instead of applying the NAT operation to the port that is to be accessed via -A. This change also prints the randomly assigned port to stdout regardless of whether --verbose mode is used (since if not then the user will have no idea which port is actually going to be NAT'd on the fwknopd side).
  • (Vlad Glagolev) Submitted an OpenBSD port for fwknop-2.0.4, and this has been checked in under the extras/openbsd/fwknop-2.0.4 directory.
  • (Shawn Wilson) Added better SPA source IP logging for various fwknopd logging messages. This helps to make it more clear why certain SPA packets are rejected from some systems.
  • [client] Added --get-hmac-key to allow HMAC keys to be acquired from the specified file similarly to the --get-key option. This is a convenience only, and the fwknop rc file feature should be used instead since it is far more powerful.

Coverity Static Analysis and Open Source Software

Coverity Scan A few months ago Coverity announced that they would grant open source projects access to their "Scan" static analysis tool for free. This was certainly a bold move, and one that has the potential to improve the security landscape as more projects take advantage of it. The first time I had encountered Coverity was several years ago when Linux kernel ChangeLog entries were referencing Coverity's Scan tool as having found various classes of bugs in the kernel sources. This impressed me because the Linux kernel source code is not the easiest code to grok, and at the same time it is extremely widely deployed and audited so bugs get reported and fixed. Andrew Morton, second in command of the kernel, has pegged the time required to become proficient as a kernel developer at about one year of full time dedicated effort for an experienced systems programmer. Hence, any automated static analysis infrastructure that is good enough to find kernel bugs and have it credited in the kernel ChangeLog is a strong vote of confidence to my mind. Here is a recent instance of Coverity finding a NULL pointer dereference bug in Infiniband driver code for example.

Even though Coverity started being used by the Linux kernel development community circa 2005, Coverity was expensive and therefore not generally available to open source projects that typically do not have access to significant resources. Today, that has all changed with Coverity's open source initiative, and the community is better off. Simultaneously, Coverity also gets a benefit in terms of free press as more open source projects start using Coverity Scan. As of this writing, over 300 projects have signed up including fwknop, and Coverity maintains a list of these here. OpenVPN and OpenSSL are participating, but it doesn't look like OpenSSH is yet and I (for one) hope they do eventually.

Here is an example of a bug that Coverity found in the fwknop client:

index fea3679..2532669 100644 (file)
@@ -552,6 +552,7 @@ send_spa_packet_http(const char *spa_data, const int sd_len,
                     "[-] proxy port value is invalid, must be in [%d-%d]",
                     1, MAX_PORT);
+                free(spa_data_copy);
                 return 0;

This is a classic memory leak bug in the fwknop client that is only triggered when an invalid port number is specified from the command line when sending SPA packet data through an HTTP proxy (this is not a widely used method for SPA packet transmission, but it is supported by the fwknop client along with other transmission modes). The spa_data_copy variable points to heap allocated memory that wasn't free()'d after the error condition was detected and before returning from the send_spa_packet_http() function call. Coverity was able to spot this bug through static analysis, and the diff output above shows the corresponding fix. In terms of other bugs found by Coverity, they have all been fixed in the current fwknop sources in preparation for the 2.5 release. A -pre release is available through the new Github releases feature, and will likely become 2.5 within a week: fwknop-2.5-pre3.tar.gz.

This same bug could also have been caught at run time with the excellent valgrind "memcheck" tool, but the invalid port number error would have had to be triggered in order for valgrind to detect the memory leak. While the fwknop test suite is fairly comprehensive and can automatically run every test underneath valgrind to help with the detection of memory leaks, not every error condition is triggered by the test suite and hence static analysis is quite important. At the same time, run time analysis is also important to pick up leaks and other problems that static analysis techniques may miss, so running valgrind will continue to be a critical test suite component.

Now, on the vulnerability research side, there are implications for vulnerability discovery. That is, source control changelogs that mention Coverity or other static analysis tools can help malicious actors find potentially exploitable code. Here is an example Github search that shows all issues where Coverity is mentioned. Is this a bad thing? Well, in a word I would say "no". The exploit community is constantly looking for ways to exploit software regardless of whether static analysis tools are used by project developers. The real problem is that bugs are frequently never discovered or fixed in the first place. Further, keeping bugs under wraps in a vein effort to hope they won't be found has never worked, and this is true regardless of whether a project is open source or not (binary patches released by Microsoft are reversed engineered to figure out how to exploit unpatched code for example). What we need is better, more security conscious software engineering - not imagined bug secrecy. Consistent, advertised usage of static analysis tools helps to drive this. In addition, static analysis tools should be seen as a piece of the bug hunting puzzle and be used in conjunction with other standard techniques such as dynamic analysis, unit tests, and good old fashioned code reviews.

Finally, in terms of Coverity's own competition such as Veracode, Klocwork, or Checkmarx, it seems to me that Coverity has made a shrewd move by offering code analysis to open source projects for free. They will continue to make important inroads into the OS community (which has become quite powerful over the past, say, 15 years and longer), and will most likely see a gain on the commercial side as a result.

The bottom line is that Coverity has an excellent static analysis product, and if you run or contribute to an open source project written in C/C++, you should be using Coverity Scan. It will likely find bugs that can certainly have security implications in your code. Even if Coverity doesn't turn up any bugs, additional verification is always a good thing - especially for security projects.

ShmooCon 2013 Talk on fwknop

ShmooCon 2013 Update 03/24/2013: All presentation videos have been uploaded to, and here is the full video for the talk discussed below: video.

This past weekend at ShmooCon 2013 I gave a talk entitled "Generalized Single Packet Authorization for Cloud Computing Environments" (slides, video demo), and in this blog post I thought it appropriate to give an introduction to the talk in blog format. The main point of the talk was to show that it is completely feasible to integrate SPA as implemented by fwknop with Cloud Computing environments that conform to the Infrastructure as a Service (IaaS) model. In particular, it was shown that SPA can be used to protect and access both SSH and RDP services running on EC2 instances within a Virtual Private Cloud (VPC) in Amazon's AWS environment. Accessing SSHD via SPA by itself is nothing new and has been a primary use case supported by fwknop for a long time, but maintaining a default-drop packet filtering stance for RDP is a different story due to the fact that fwknop does not support a Windows firewall. However, through the use of both SNAT and DNAT capabilities within iptables, the fwknop daemon is able to convert an Ubuntu EC2 instance running within Amazon's Cloud into an "SPA gateway" that is able to then allow access to any service running on any other system within the VPC network - including RDP on a separate Windows host. Further, such access is granted through a single externally routable Amazon Elastic IP that is associated with the Ubuntu system. The Windows host doesn't even need to have a default route back out to the Internet - it always just sees incoming RDP connections from the Ubuntu host even though they really originate out on the Internet from whatever system is allowed via the SPA protocol. Here is a graphic to illustrate this scenario: fwknop + Amazon integration If you want a video demonstration of this all in action, this demo was given during the ShmooCon talk:

Now, for more explanation, the following systems are depicted in the graphic above:

  • The SPA client system that runs the fwknop client to send SPA packets and from which SSH and RDP connections originate.
  • The Amazon border firewall.
  • An Ubuntu host on the VPC network where the fwknop daemon runs.
  • A Windows system on the VPC network where RDP lives.
At the same time, the Ubuntu VPC host is also running a webserver. This is to illustrate that everyone out on the Internet can scan the externally routable Elastic IP ( in this case) associated with the Ubuntu host and see that Apache is listening on port 80. (Note that this IP is not currently associated with my Amazon account.) But, no other services are visible to a scanner on the Ubuntu system because iptables is deployed in a default-drop stance for everything except for port 80. Even so, fwknop is able to create access through the iptables policy for SSH and RDP through the Ubuntu system over port 80 (i.e. SPA "ghost services"). These two scripts that are shown in the demo video illustrate gaining access to SSH and RDP:
$ cat
#!/bin/sh -x

fwknop -A tcp/80 --server-port 40001 -R -D
sleep 2
ssh -p 80 -i ~/.ssh/cdyne-vpc-testing.pem ubuntu@

$ cat
#!/bin/sh -x

fwknop -A tcp/80 --server-port 40001 -N,3389 -R -D
sleep 2
rdesktop -u Administrator

To see all of the above in action just watch the video. Hopefully the talk itself will be posted online soon, and when it is I'll post a link. More information can also be found in the talk slides.

Finally, I'd like to thank the ShmooCon organizers for putting on a great conference as usual, and I'm looking forward to next year.

Software Release - fwknop-2.0.4

fwknop-2.0.4 released The 2.0.4 release of fwknop is available for download. This release includes new support for fuzzing the SPA protocol powered by the perl FKO module along with a set of patches designed to break libfko in subtle ways, and also includes several bug fixes and small enhancements. Among the most important of these include a new check for fwknop chains on iptables firewalls after the initial fwknopd start up (so the fwknop chains can be deleted out from under the fwknop daemon without affecting SPA operations), and a new 'upstart' config for systems running the upstart daemon. Many other changes were contributed by the open source community, and these are acknowledged in the complete fwknop-2.0.4 ChangeLog below:

  • [client] Misc fixes and the addition of save_args and last command (.fwknop.last) support on the Windows platform.
  • [client] Fixed bug in username determination code where a valid value could be overrwritten in certain circumstances.
  • [server] Added upstart config at extras/upstart/fwknop.conf. This allows the fwknopd to easily be managed with upstart via commands like "service fwknop start" and "service fwknop stop".
  • [server] (Vlad Glagolev) Submitted a patch to fix ndbm/gdbm usage when --disable-file-cache is used for the autoconf configure script. This functionality was broken in be4193d734850fe60f14a26b547525ea0b9ce1e9 through improper handling of #define macros from --disable-file-cache.
  • [server] (Vlad Glagolev) Submitted a patch to fix command exec mode under SPA message type validity test. Support for command exec mode was also added to the test suite.
  • (Vlad Glagolev) Submitted an OpenBSD port for fwknop-2.0.3, and this has been checked in under the extras/openbsd/ directory.
  • [server] Bug fix to allow GPG_ALLOW_NO_PW to result in not also having to specify a Rijndael key.
  • [Android] Added new .properties files to allow the fwknop client to build and function properly on the latest Android release (4.1.2).
  • [client] Added '-P udpraw' to allow the client to send SPA packets over UDP with a spoofed source IP address. This is in addition to the original 'tcpraw' and 'icmp' protocols that also support a spoofed source IP.
  • [libfko] Bug fix to check b64_decode() return value to ensure that non-base64 encoded data is never used. Even though other validation routines checked decoded results, it is important to discard invalid data as early as possible. Note too that such invalid data would only be provided to b64_decode() after proper decryption, so the client must provide authentic SPA data.
  • [libfko] Added validation of NAT access strings in the various NAT modes.
  • [libfko] Restricted usernames embedded in SPA packets to be alpha-numeric along with "-" chars.
  • [client] (Franck Joncourt) Contributed a patch to allow the fwknop client to be stopped during the password entry prompt with Ctrl-C before any SPA packet is sent on the wire.
  • [client+server] Applied patch from Franck Joncourt to remove unnecessary chmod() call when creating client rc file and server replay cache file. The permissions are now set appropriately via open(), and at the same time this patch fixes a potential race condition since the previous code used fopen() followed by chmod().
  • [server] Bug fix to accept SPA packets over ICMP if the fwknop client is executed with '-P icmp' and the user has the required privileges.
  • [test suite] Applied patch from Franck Joncourt to have the perl FKO module link against libfko in the local directory (if it exists) so that it doesn't have to have libfko completely installed in the /usr/lib/ directory. This allows the test suite to run FKO tests without installing libfko.
  • [test suite] Significant update to include a set of fuzzing SPA packets that are built using a patched version of libfko. These packets are located in the test/fuzzing/bogus_spa_packets file, and are designed to ensure proper validation of SPA packet data. This validation is performed in --enable-perl-module-checks mode via the perl FKO module.
  • [client] Added --icmp-type and --icmp-code arguments so the user can control the icmp type/code combination for spoofed SPA packets ('-P icmp') mode.
  • [client] Updated default TTL value to 64 for spoofed SPA packets. This is closer to more OS default TTL values than the previous 255.
  • Updated build CFLAGS and LDFLAGS to conform to the Debian hardening-includes file for PIE support (e.g. '-fPIE' for CFLAGS and '-fPIE -pie' for LDFLAGS).
  • [test suite] For GnuPG tests that require a passphrase associated with a gpg key, added a pinentry check to see if the local gpg engine requires it. If so, the gpg test that require a key are excluded since.
  • [server] Added a new '--pcap-file <file>' option to allow pcap files to be processed directly by fwknopd instead of sniffing an interface. This feature is mostly intended for debugging purposes.
  • [server] Added chain_exists() check to SPA rule creation so that if any of the fwknop chains are deleted out from under fwknopd they will be recreated on the fly. This mitigates scenarios where fwknopd might be started before a system level firewall policy is applied due to init script ordering, or if an iptables policy is re-applied without restarting fwknopd.
The complete fwknop-2.0.4 ChangeLog can also be found here via the fwknop gitweb interface.

Tutorial on Single Packet Authorization with fwknop

Single Packet Authorization: The fwknop Approach One thing that has been requested a few times is a tutorial on how to use fwknop, so here it is: "Single Packet Authorization: A Comprehensive Guide to Strong Service Hardening with fwknop". This document attempts to cover most usages of fwknop to wrap an SPA layer around arbitrary services, though some less used scenarios such as SPA over Tor and SPA packet spoofing are covered as well. Please contact me if you find any errors, or if you would like to see material added on a particular fwknop use case. The complete table of contents appears below:

Single Packet Authorization: The fwknop Approach

Single Packet Authorization: The fwknop Approach There are many different port knocking implementations out there, and a few that take things to the next level with Single Packet Authorization. All PK/SPA implementations have three primary goals: 1) the deployment of a default-drop firewall policy for a service (such as SSHD) in order to protect against scans, potential vulnerabilities, and brute force password guessing attempts, 2) the passive monitoring of specially constructed authentication information from the PK/SPA client, and 3) the dynamic reconfiguration of the firewall to allow temporary access to the concealed service.

The PK/SPA server never acknowledges any data from the client, so it is infeasible to scan not only for any properly implemented PK/SPA server, but also for any protected service. Effectively, services become invisible to the Internet - except for those that can produce the necessary authentication information, or those that are in a privileged position to sniff traffic between an authenticated PK/SPA client and the target service.

Users of firewalls find value in the idea that traffic to a service can be blocked from all but a few pre-defined networks according to the firewall policy. Few people question whether this is valuable from a security perspective - firewalls generally enhance security (the occasional firewall vulnerability not withstanding). The PK/SPA strategy extends the notion of filtering traffic for a set of services by adding a lightweight crypto layer to allow temporary access from networks that cannot be anticipated when the firewall policy is written. This provides concealment by default, and the SPA strategy asserts that there is value in this. This can apply to all sorts of services from SSHD and OpenVPN to mail protocols like POP and IMAP and even to HTTP. Anyone scanning for services to target will never see a service protected by SPA.

If the above describes unifying aspects of PK/SPA software, this is where the similarities in PK/SPA software ends.

Below are the design choices that guide fwknop development. Other PK/SPA software make different design choices, and whether you prefer fwknop vs. another implementation depends at least partially on whether you agree with the following:

  • No heavyweight interpreted languages:
    Many people don't want to install perl, python, or ruby on a firewall or network gateway device. Such languages are large and complex, and are implemented by similarly complex binaries that are usually themselves written in a language like C. Firewalls are frequently stripped down systems that are designed to just filter network traffic, provide administrative interfaces and sometimes VPN services, and not do much else. Both the fwknop client and server are entirely written in C - there is no requirement for perl, python, or any other interpreted language.

  • Support embedded devices:
    A consequence of fwknop being developed in C is that it only uses minimal system resources and can therefore support embedded devices that don't have a lot of computing power or main memory. For example, where the older perl version of fwknop could not run on Linksys routers supported by OpenWRT, these same routers can run the newer C version of fwknop.

  • Don't require admin access to run the SPA client:
    There are many computing environments where users don't have privileged accounts. This should not present a barrier to using the SPA client. In fwknop, SPA packets are (by default) sent over a regular UDP socket and therefore require no special privileges.

  • Don't require the manipulation of raw packet headers:
    There are some SPA implementations that communicate information within specially modified fields within IP or TCP headers. Such manipulation requires that the user have the ability to acquire a raw socket from the OS, and this requires admin level privileges. In addition, depending on how the SPA client manipulates packet headers when building an SPA packet, other monitoring systems such as an IDS or a passive OS fingerprinter may produce event data that unnecessarily calls attention to the SPA communications. This is not to say that that is impossible to detect SPA packets generated by fwknop - it is just that a monitoring system is more likely to flag communications that involve manipulated packet headers than to generate an event for packets produced by fwknop. For example, intrusion detection systems track TCP connections, and spoofed TCP ACK's that are not part of a proper connection (assuming non-asymmetric routing) may potentially be flagged. Also, manipulated TCP options fields that don't conform to OS defaults will cause an OS to appear to change under the observation of things like p0f. While sometimes this is an expected behavior such as if a VM is being run or a system is actually a NAT device with other systems behind it, there are plenty of deployment scenarios where this is not expected. Addressing this issue at length could be a separate blog post in itself.

  • Don't trust the IP header:
    Any SPA implementation that trusts the source IP address in the IP header of the SPA packet is vulnerable to a MITM attack. An inline device that can intercept an SPA packet can hold the original packet and retransmit it but with the source IP changed to whatever the attacker wants. If the SPA server relies on the source IP in the IP header, then it has no way to know that it isn't legitimate. If an SPA implementation is going to go to the trouble of leveraging cryptography - certainly important since replay attacks among other problems can't be prevented without it - then the IP should be made part of an encrypted payload. This is exactly what fwknop does with the -a or -R arguments on the fwknop command line. An attacker can intercept an SPA packet produced by fwknop, change the source IP and retransmit, but SPA server will only allow access to the IP that was originally encrypted within the SPA payload.

  • Support server-side NAT:
    There are plenty of networks with a border firewall where a remote user actually wants access to a service that is running on an internal system and not on the firewall itself. Typical ways of accessing such a service involve running VPN software, but with an SPA implementation that can manipulate NAT rules on the border firewall it is possible to transparently grant SPA users access to internal services through the firewall. With fwknop, the capability to leverage server-side NAT has been built in for a long time.

  • Support multiple firewalls:
    Because fwknop does not rely on specialized logging infrastructure or link against libraries that are tied to one firewall architecture or another, it can easily support multiple firewalls just by executing the local firewall admin command line interface. It currently supports iptables on Linux, ipfw on FreeBSD and Mac OS X, and pf on OpenBSD.

  • Minimize library dependencies:
    Given the design decisions made by fwknop above, it is important to minimize library dependencies and to audit the source code. Here are all library dependencies in the fwknopd daemon including GnuPG support (this is optional):
    $ ldd ./server/.libs/fwknopd =>  (0x00007ffeebf820e0) => /usr/lib/ (0x00007f1a6ae930e0) => /usr/lib/x86_64-linux-gnu/ (0x00007e1a6a85c0e0) => /lib/x86_64-linux-gnu/ (0x00007e1a6a49e0a0) => /usr/lib/ (0x00007f1aeaee800e)
        /lib64/ (0x00007e1a6aede0e0) => /lib/x86_64-linux-gnu/ (0x0000ef1a6a06e0e0)
    The libfko library is developed by the fwknop project for SPA encryption, decryption, and validation. libpcap is a standard C library that is leveraged by a lot of security projects including intrusion detection systems and more. The gpgme libraries are only used if you want GnuPG support, and the remaining libraries are standard system libraries that typical Linux binaries need. That's it. One may certainly question the use of libpcap, and in defense of this choice note that: 1) fwknopd by default does not put the sniffing interface into promiscuous mode, 2) fwknopd by default only pays attention to UDP packets to port 62201, 3) if you examine Metasploit you'll see that there only a few exploits for pcap-based software and they target what rides on top of libpcap vs. libpcap itself, and 4) even if there is a vulnerability in libpcap, the exploit model is different than it is for normal server software that you can easily scan for, and being different in this game is good.

  • Support both symmetric and asymmetric ciphers:
    Some users prefer the security properties that asymmetric crypto systems have over those provided by symmetric algorithms like Rijndael, or vice versa. fwknop supports both Rijndael and GnuPG for SPA encryption/decryption.

  • Support HMAC in the encrypt-then-authenticate model:
    Authenticated encryption with an HMAC is supported by fwknop as of the 2.5 release for both symmetric and asymmetric encryption modes. The implementation is careful to apply an HMAC to SPA packets according to the encrypt-then-authenticate model which provides strong resistance to certain kinds of cryptanalytic attacks. In addition, on the fwknopd server side, verification of the HMAC is a more simplistic operation than sending data through a set of decryption routines, so this provides an additional security benefit by reducing the complexity of code the malicious traffic can interact with. This is particularly true for SPA packets that are encrypted with GnuPG.

Software Release - fwknop-2.0.3

fwknop-2.0.3 released The fwknop-2.0.3 release of fwknop is available for download. This is an important release that tightens up the code in several areas based on excellent research performed by Fernando Arnaboldi of IOActive. I wish to thank Fernando for this work, and also Erik Gomez of IOActive as well for making this research possible. Fernando's research turned up potential DoS and/or code execution vulnerabilities in the fwknopd server for malicious authenticated clients, insecure filesystem permissions handling, and more. All of these issues have been fixed in the 2.0.3 release.

Here is the complete fwknop-2.0.3 ChangeLog:

  • [server] Fernando Arnaboldi from IOActive found several DoS/code execution vulnerabilities for malicious fwknop clients that manage to get past the authentication stage (so a such a client must be in possession of a valid access.conf encryption key). These vulnerabilities manifested themselves in the handling of malformed access requests, and both the fwknopd server code along with libfko now perform stronger input validation of access request data. These vulnerabilities affect pre-2.0.3 fwknop releases.
  • [server] Fernando Arnaboldi from IOActive found a condition in which the server did not properly validate allow IP addresses from malicious authenticated clients. This has been fixed with stronger allow IP validation.
  • [client+server] Fernando Arnaboldi from IOActive found that strict filesystem permissions for various fwknop files are not verified. Added warnings whenever permissions are not strict enough, and ensured that files created by the fwknop client and server are only set to user read/write.
  • [client] Fernando Arnaboldi from IOActive found a local buffer overflow in --last processing with a maliciously constructed ~/ file. This has been fixed with proper validation of arguments.
  • [server] Fernando Arnaboldi from IOActive found several conditions in which the server did not properly throw out maliciously constructed variables in the access.conf file. This has been fixed along with new fuzzing tests in the test suite.
  • [test suite] Added a new fuzzing capability to ensure proper server-side input validation. Fuzzing data is constructed with modified fwknop client code that is designed to emulate malicious behavior.
  • Fixed RPM builds by including the $(DESTDIR) prefix for uninstall-local and install-exec-hook stages in
The fwknop-2.0.3 ChangeLog can also be found here via the fwknop gitweb interface.
Next »