Michael Rash, Security Researcher

System Administration    [Summary View]

HTTP Response Times for

smokeping I'm a big fan of the Smokeping project developed by Tobi Oetiker, and I use it to graph ICMP, DNS, and HTTP round trip times for my local internet connections and also for Since the beginning of 2010, I've switched around where was hosted from a system at a hosting provider in Canada to a system running on my home network (serviced by a Verizon FiOS connection). The drop in RTT's in mid-February was expected and quite large - going from an average of around 200ms down to about 20ms. Then, in June, I switched to a different Verizon DSL connection and moved the webserver system to a newer Verizon FiOS connection at a different location, and the RTT's went up a bit to about 50ms. Here is the graph: 400 days

Analyzing a Trac SPAM Attempt

Spam Attempts Through Trac One of the best web interfaces for visualizing Subversion repositories (as well as providing integrated project management and ticketing functionality) is the Trac Project, and all Cipherdyne software projects use it. With Trac's success, it also becomes a target for those that would try to subvert it for purposes such as creating spam. Because I deploy Trac with the Roadmap, Tickets, and Wiki features disabled, my deployment essentially does not accept user-generated content (like comments in tickets for example) for display. This minimizes my exposure to Trac spam, which has become a problem significant enough for various spam-fighting strategies to be developed such as configuring mod-security and a plugin from the Trac project itself.

Even though my Trac deployment does not display user-generated content, there are still places where Trac accepts query strings from users, and spammers seem to try and use these fields for their own ends. Let's see if we can find a few examples. Trac web logs are informative, but sifting through huge logfiles can be tedious. Fortunately, simply sorting the logfiles by line length (and therefore by web request length) allows many suspicious web requests to bubble up to the top. Below is a simple perl script that sorts any ascii text file by line length, and precedes each printed line with the line length followed by the number of lines equal to that length. That is, not all lines are printed since the script is designed to handle large files - we want the unusually long lines to be printed but the many shorter lines (which represent the vast majority of legitimate web requests) to be summarized. This is an important feature considering that at this point there is over 2.5GB of log data specifically from my Trac server.

$ cat
#!/usr/bin/perl -w
# prints out a file sorted by longest lines
# $Id: 1739 2008-07-05 13:44:31Z mbr  $

use strict;

my %url       = ();
my %len_stats = ();
my $mlen = 0;
my $mnum = 0;

open F, "< $ARGV[0]" or die $!;
while (<F>) {
    my $len = length $_;
    $url{$len} = $_;
    $mlen = $len if $mlen < $len;
    $mnum = $len_stats{$len}
        if $mnum < $len_stats{$len};
close F;

$mlen = length $mlen;
$mnum = length $mnum;

for my $len (sort {$b <=> $a} keys %url) {
    printf "[len: %${mlen}d, tot: %${mnum}d] %s",
            $len, $len_stats{$len}, $url{$len};

exit 0;
To illustrate how it works, below is the output of the script used against itself. Note that at the top of the output the more interesting code appears whereas the most uninteresting code (such as blank lines and lines that contain closing "}" characters) are summarized away at the bottom:
$ ./
[len: 51, tot: 1] # $Id: 1739 2008-07-05 13:44:31Z mbr  $
[len: 50, tot: 1]     printf "[len: %${mlen}d, tot: %${mnum}d] %s",
[len: 48, tot: 1]             $len, $len_stats{$len}, $url{$len};
[len: 44, tot: 1] # prints out a file sorted by longest lines
[len: 43, tot: 1] for my $len (sort {$b <=> $a} keys %url) {
[len: 37, tot: 1]         if $mnum < $len_stats{$len};
[len: 34, tot: 1]     $mlen = $len if $mlen < $len;
[len: 32, tot: 1] open F, "< $ARGV[0]" or die $!;
[len: 29, tot: 1]     $mnum = $len_stats{$len}
[len: 25, tot: 1]     my $len = length $_;
[len: 24, tot: 1]     $len_stats{$len}++;
[len: 22, tot: 2] $mnum = length $mnum;
[len: 21, tot: 1]     $url{$len} = $_;
[len: 20, tot: 2] my %len_stats = ();
[len: 19, tot: 1] #!/usr/bin/perl -w
[len: 14, tot: 3] while (<F>) {
[len: 12, tot: 1] use strict;
[len:  9, tot: 1] close F;
[len:  8, tot: 1] exit 0;
[len:  2, tot: 5] }
[len:  1, tot: 6] 
Now, let's execute the script against the trac_access_log file and look at one of the longest web requests. (The script was able to reduce the 12,000,000 web requests in my Trac logs to a total of 610 interesting lines.) This particular request is 888 characters long, but there were some other similar suspicious requests that had over 4,000 characters that are not displayed for brevity: [len: 888, tot: 1] - - [02/Mar/2008:00:30:17 -0500] "GET /trac/fwsnort/anydiff?new_path=%2Ffwsnort%2Ftags%2Ffwsnort -1.0.3%2Fsnort_rules%2Fweb-cgi.rules&old_path=%2Ffwsnort%2Ftags%2F fwsnort-1.0.3%2Fsnort_rules%2Fweb-cgi.rules&new_rev=http%3A%2F%2Ff map.html%0A& HTTP/1.1" 200 3683 "-" "User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.2)" My guess is that the above request is a bot that is trying to do one of two things: 1) force Trac to accept the content in the request (which contains a bunch of links to pages like "" - note that I altered the domain so as to not legitimize the original content) and display it for other Trac users or to search engines, or 2) force Trac itself to generate web requests to the provided links (perhaps as a way to increase hit or referrer counts from domains - like mine - that are not affiliated with the spammer). Either way, the strategy is flawed because the request is against the Trac "anydiff" interface which doesn't accept user content other than svn revision numbers, and (at least in Trac-0.10.4) such requests do not cause Trac to issue any external DNS or web requests - I verified this with tcpdump on my Trac server after generating similar requests against it.

Still, in all of my Trac web logs, the most suspicious web requests are against the "anydiff" interface, and specifically against the "web-cgi.rules" file bundled within the fwsnort project. But, the requests never come from the same IP address, the "anydiff" spam attempts never hit any other link besides the web-cgi.rules page, and they started with regularity in March, 2008. This makes a stronger case for the activity coming from bot that is unable to infer that its activities are not actually working (no surprise there). Finally, I left the original IP address of the web request above intact so that you can look for it in your own web logs. Although is not listed in the Spamhaus DNSBL service, a rudimentary Google search indicates that has been noticed before by other sites as a comment spammer.

Interfacing VIM with GnuPG Encrypted Files

Digg VIM + GnuPG Interfacing VIM with GnuPG Encrypted Files Over the years the editor I have become the most familiar with is vim. It provides features that allow the software development process to go smoothly, such as split screens, syntax highlighting, integration with cscope tags, function folding, and more. By applying configuration directives to your ~/.vimrc file, you can instruct vim to perform some nice preprocessing functions against files. This blog post illustrates a ~/.vimrc tweak (originally from Wouter Hanegraaff) that allows vim to leverage GnuPG to decrypt a previously encrypted file, allow edits to be made, and then re-encrypted before it is written back to disk. Here is the section of the .vimrc file to allow such transparent encryption and editing: " Transparent editing of gpg encrypted files.
" By Wouter Hanegraaff <>
augroup encrypted
" First make sure nothing is written to ~/.viminfo while editing
" an encrypted file.
autocmd BufReadPre,FileReadPre *.gpg set viminfo=
" We don't want a swap file, as it writes unencrypted data to disk
autocmd BufReadPre,FileReadPre *.gpg set noswapfile
" Switch to binary mode to read the encrypted file
autocmd BufReadPre,FileReadPre *.gpg set bin
autocmd BufReadPre,FileReadPre *.gpg let ch_save = &ch|set ch=2
autocmd BufReadPost,FileReadPost *.gpg '[,']!gpg --decrypt 2> /dev/null
" Switch to normal mode for editing
autocmd BufReadPost,FileReadPost *.gpg set nobin
autocmd BufReadPost,FileReadPost *.gpg let &ch = ch_save|unlet ch_save
autocmd BufReadPost,FileReadPost *.gpg execute ":doautocmd BufReadPost " . expand("%:r")
" Convert all text to encrypted text before writing
autocmd BufWritePre,FileWritePre *.gpg '[,']!gpg --default-recipient-self -ae 2>/dev/null
" Undo the encryption so we are back in the normal text, directly
" after the file has been written.
autocmd BufWritePost,FileWritePost *.gpg u
You can combine this vim tweak with gpgdir to maintain recursively encrypted directories, and just edit the files directly. For example, the following sequence of commands shows the creation of an encrypted file and how vim then interfaces with GnuPG to allow transparent editing: $ cat > somefile
private data
more private data
$ gpg -e somefile
$ wipe somefile
Okay to WIPE 1 regular file ? (Yes/No) yes
Operation finished.
1 file wiped and 0 special files ignored in 0 directories, 0 symlinks removed but not followed, 0 errors occured.
$ ls -l somefile.gpg
-rw-r--r-- 1 mbr mbr 618 2008-02-17 01:52 somefile.gpg
$ vim somefile.gpg
"somefile.gpg" [noeol] 3L, 618C

You need a passphrase to unlock the secret key for
user: "Michael Rash <>"
2048-bit ELG-E key, ID 1234ABCD, created 2007-05-01 (main key ID ABCD1234)

Enter passphrase:

<apply edits now, and hit :wq >

$ ls -l somefile.gpg
-rw-r--r-- 1 mbr mbr 932 2008-02-17 01:55 somefile.gpg
As you can see from the output above, the file was modified (and the original file somefile was deleted using wipe).

Finally, the 1.6 (UPDATE: actually, the 1.7 release is available as of 02/18/2008) release of gpgdir is ready for download. This is a bugfix release that restores the exclusion of previously encrypted files from the file selection process, and includes a few minor enhancements as well. Here is the ChangeLog:
  • Bugfix to not include previously encrypted files (i.e. those with a .gpg extension) in the encryption/decryption file list. This bug was introduced in gpgdir-1.5 when a change was made to ignore ascii-armored files.
  • Added added LC_ALL=C locale setting for the script (this should help to ensure gpgdir is properly installed on most systems). Two new command line arguments --LC_ALL and --no-LC_ALL also allow the locale setting to be changed or not used at all.
  • Added --Exclude-mod-regex option to the script so that it is possible to force the exclusion of perl modules that gpgdir would normally install. This is useful for ensuring that gpgdir references perl modules that are already installed in the system perl library tree instead of using those that are installed in /usr/lib/gpgdir.
  • Updated to display command line usage warnings without automatically displaying the entire usage() page (which is quite long).

Deploying the Trac Timeline and Browse Source Features

Deploying the Trac Timeline and Browse Source Features The Trac Project provides an excellent web interface for project documentation, ticket-based project management, and visualization of source code (from a Subversion repository). All of the Cipherdyne open source projects use Trac as the code display mechanism, and allow users to readily see how the projects are developed over time. However, the Cipherdyne projects are not large software engineering efforts with many dedicated developers, and I prefer not to allow my webserver to accept user registrations and tickets even if it is through Trac. The project mailing lists as well as personal email correspondence has been (so far) completely sufficient for bug reporting, and development tasks are tracked within TODO files in each of the top level source directories for each project. Trac also offers a Wiki, but I write all of my own documentation for my projects and I accept enhancements via email as well.

So, why the blog post? Well, it is not immediately obvious how to enable just the Timeline, Browse Source, and Search features in Trac, and disable the remaining features such as the Roadmap, Tickets, and the Wiki. (All of these features can be seen in the Trac site in the navigation bar off to the right.) It turns out that Trac is database driven, and disabling these features can be accomplished from the command line as follows with the trac-admin command: $ trac-admin /path/to/trac_directory permission remove anonymous TICKET_CREATE TICKET_MODIFY TICKET_VIEW ROADMAP_VIEW REPORT_VIEW MILESTONE_VIEW REPORT_SQL_VIEW WIKI_CREATE WIKI_MODIFY WIKI_VIEW Also, you will need to set the default_handler variable in the conf/trac.ini file to BrowserModule instead of WikiModule. Using the above command, the Trac navigation bar only includes the Timeline, Browse Source, and Search features as seen here, and this is a valuable configuration for small open source projects. However, if you would like additional functionality in Trac to be enabled for the Cipherdyne projects please email me; perhaps there are benefits here that would justify the change.