systhread.net

Jul 2007

An ip_conntrack_max Threshold Script

Iptables is great, easy to setup and generally worry free once it is all configured. Except of course if one forgets to keep an eye on the state machine. ip_conntrack does just what is says; tracks current connections (with a time lag of course). There is a variable in /proc/sys/net/ipv4 labelled ip_conntrack_max which also means what it says... when ip_conntrack_max is hit (or even close to being hit): bad things can happen. Solution: write a script to keep an eye on ip_conntrack. Note this only helps with kernel 2.4. In the 2.6 kernel a variety of netfilter parameters can be changed [3]. Of course there is always the first draft and refinements to follow.

The Problem

When ip_conntrack is getting close to ip_conntrack_max iptables starts dropping valid packets. Dropped packets can be a minor annoyance in some configurations or an all out disaster in others. To top it off, systems retransmitting simply exacerbated the problem even further [ 1 ].

A Solution

One possible solution is to monitor the status of connections. For instance, periodically getting the number of current connections:

wc -l /proc/net/ip_conntrack | mail -s "Current IP Connections" \
        someadmin@somemail.net

Of course periodic mails are heavy handed. What would make more sense is once a reasonable threshold is calculated; check to make sure the threshold is not approached.

A Perl Script

In the example a simple Perl script is used. Before getting into the actual checks a look at interrupt handling and some helper functions.

Signals

#!/usr/bin/env perl

use strict;

$SIG{'INT' } = 'interrupt';     $SIG{'QUIT'} = 'interrupt';
$SIG{'HUP' } = 'interrupt';     $SIG{'TRAP'} = 'interrupt';
$SIG{'ABRT'} = 'interrupt';     $SIG{'STOP'} = 'interrupt';

#------------------------------------------------------------------------------
# Simple interrupt handling
#------------------------------------------------------------------------------
sub interrupt {
    my($sig) = @_;
    die $sig;
    die;
}

Pretty standard stuff, the shebang, load up the strict pragma then a simple (and no frills) routine to handle certain interrupts.

File Loader

There will be a need to load several files within the script, following the rule of never do anything more than once unless you have to a short routine to load up a file and return the contents in an array to the caller:

#------------------------------------------------------------------------------
# Generic file loader
#------------------------------------------------------------------------------
sub load_file {
    my ($file) = shift;
    my @flist;

    open(FILE, $file) or die "Unable to open logfile $file: $!\n";
    @flist = <FILE>;
    close FILE;

    return(@flist);
}

Figuring out RAM Limits

How much ram can iptables take before you need more? Tough question and there is no precise answer. No doubt about it, iptables and the connection tracking will take RAM. 70% is used - remember; physical memory not virtual memory. Each connection requires about 1/16384 of RAM [2]. In the following sub routine physical memory is read from /proc/meminfo and the result * .7 * .015 is returned:

#------------------------------------------------------------------------------
# Return max mem we are willing to use
#------------------------------------------------------------------------------
sub get_mem_max {
    my @kmeminfo = load_file("/proc/meminfo");
    my $ram_total = @kmeminfo[3];

    $ram_total =~ s{MemTotal:}{};
    $ram_total =~ s{kB}{};
    $ram_total =~ s{MB}{};

    return (.7 * $ram_total) * .06;
}

The Actual Check

In the check itself there are several steps to take, read in the current maximum, read in the current connections, use .6 (60%) as the ip_conntrack threshold, make sure there is room in RAM (remembering there is some fudge factor) and either:

1. Double the maximum
2. Error Break

#------------------------------------------------------------------------------
# Check the current connections situation...
#------------------------------------------------------------------------------
sub check_ip_conntrack {
    my $ip_conntrack_max = `cat /proc/sys/net/ipv4/ip_conntrack_max`;
    my $ip_conntrack_cur = `wc -l /proc/net/ip_conntrack`;
    my $hi_water_mark = ($ip_conntrack_max * .6);
    my $ip_conntrack_hard_limit = get_mem_max();

    if ($ip_conntrack_cur >= $hi_water_mark) {
        my $new_value = ($hi_water_mark * 2);

        if (($new_value * 65000) >= $ip_conntrack_hard_limit) {
            print "Error! IP CONNTRACK HAS REACHED THE 70% of RAM HIMARK!\n";
            exit 0;
        } else {
            system("echo $new_value > /proc/sys/net/ipv4/ip_conntrack_max");
        }
    }
}

Putting it All Together

Following is the script; draft1:

#!/usr/bin/env perl
#
# Script ----------------------------------------------------------------------
# $Id: $
#
# Description - Take care of ip_conntrack entries.
#
# Created
#   Author: jrf
#   Date:   2007/05/02
#
# Last Modified
#   $Author: $
#   $Date: $
#   $State: $
#
#------------------------------------------------------------------------------

use strict;

# Interrupts to trap
$SIG{'INT' } = 'interrupt';     $SIG{'QUIT'} = 'interrupt';
$SIG{'HUP' } = 'interrupt';     $SIG{'TRAP'} = 'interrupt';
$SIG{'ABRT'} = 'interrupt';     $SIG{'STOP'} = 'interrupt';

#------------------------------------------------------------------------------
# Simple interrupt handling
#------------------------------------------------------------------------------
sub interrupt {
    my($sig) = @_;
    die $sig;
    die;
}

#------------------------------------------------------------------------------
# Check the current connections situation...
#------------------------------------------------------------------------------
sub check_ip_conntrack {
    my $ip_conntrack_max = `cat /proc/sys/net/ipv4/ip_conntrack_max`;
    my $ip_conntrack_cur = `wc -l /proc/net/ip_conntrack`;
    my $hi_water_mark = ($ip_conntrack_max * .6);
    my $ip_conntrack_hard_limit = get_mem_max();

    if ($ip_conntrack_cur >= $hi_water_mark) {
        my $new_value = ($hi_water_mark * 2);

        if (($new_value * 65000) >= $ip_conntrack_hard_limit) {
            print "Error! IP CONNTRACK HAS REACHED THE 70% of RAM HIMARK!\n";
            exit 0;
        } else {
            system("echo $new_value > /proc/sys/net/ipv4/ip_conntrack_max");
        }
    }
}

#------------------------------------------------------------------------------
# Calculate how much memory we can gobble up for conntrack
#------------------------------------------------------------------------------
sub get_mem_max {
    my @kmeminfo = load_file("/proc/meminfo");
    my $ram_total = @kmeminfo[3];

    $ram_total =~ s{MemTotal:}{};
    $ram_total =~ s{kB}{};
    $ram_total =~ s{MB}{};

    return (.7 * $ram_total)*.06;
}

#------------------------------------------------------------------------------
# Generic file loader
#------------------------------------------------------------------------------
sub load_file {
    my ($file) = shift;
    my @flist;

    open(FILE, $file) or die "Unable to open logfile $file: $!\n";
    @flist = <FILE>;
    close FILE;

    return(@flist);
}

# No input required - just run
check_ip_conntrack();

Improvements

The first draft of the script does the job, however, it is lacking:

• The precision for calculating buckets is not done very well. It is only two decimal places. Some administrators may require up to 5 decimal places of precision. The kernel (as of this writing) does 5.
• There are two instances where Perl system() like calls are used to do operations Perl 5 can do by itself.
• Not overly important, but some of the variables could use renaming.

Assuming draft 1 is in production for stop gap - it is now time to go back and revise the script. Precision is easy to fix, just use longer expressions :

#------------------------------------------------------------------------------
# Calculate how much memory we can gobble up for conntrack
#------------------------------------------------------------------------------
sub get_mem_max {
    my @kmeminfo = load_file("/proc/meminfo");
    my $ram_total = @kmeminfo[3];

    $ram_total =~ s{MemTotal:}{};
    $ram_total =~ s{kB}{};
    $ram_total =~ s{MB}{};

    return ((.70000 * $ram_total) * (1/16384))
}

Next, fixup some variable names - the script is for ip_conntrack; no reason to state the fact over and over:

#------------------------------------------------------------------------------
# Check the maximum ip conntracks and adjust
#------------------------------------------------------------------------------
sub checkmax {
    my $max = `cat /proc/sys/net/ipv4/ip_conntrack_max`;
    my $cur = `wc -l /proc/net/ip_conntrack`;
    my $hi_water_mark = ($max * .6);
    my $hard_limit = get_mem_max();

    if ($cur >= $hi_water_mark) {
        my $new_value = ($hi_water_mark * 2);

        if (($new_value * (1/16384)) >= $hard_limit) {
            print "Error! IP CONNTRACK HAS REACHED THE 70% of RAM HIMARK!\n";
            exit 0;
        } else {
            system("echo $new_value > /proc/sys/net/ipv4/ip_conntrack_max");
        }
    }
}

Note the calculation using the bucket numbers was fixed as well. Now it is time to look at internalizing the two callouts:

...
    my $max = `cat /proc/sys/net/ipv4/ip_conntrack_max`;
    my $cur = `wc -l /proc/net/ip_conntrack`;
...

There are many ways to do the above in Perl. What is shown below is just one way, For the line count, just use the load_file() routine from another routine (just in case it is needed again in the future):

sub count {
        my $fp = shift;
        my @fp_contents = load_file($fp);
        return (scalar(@fp_contents));
}

Next up, use load_file again - just take out what we need instead of calling cat:

#------------------------------------------------------------------------------
# Check the maximum ip conntracks and adjust
#------------------------------------------------------------------------------
sub checkmax {
    my @max_file = load_file("/proc/sys/net/ipv4/ip_conntrack_max");
        my $max = $load_file[0];
        chomp($max) # get rid of the end of line
        my $cur = count("/proc/net/ip_conntrack");
...

Done. There are probably better ways to do it - but the goal is accomplished for the time being. A few obvious problems nixed and it is back in production.

Summary

The example was done in Perl. It could just has easily have been done in Bash or Python. Doesn't matter, what does matter is when a quick script has to be written the first draft will often do the job just don't forget to go back and improve it.

Footnotes

2. I do not know, in the particular configuration and network I was working on if the problem became one of an n(0) magnitude but it is certainly possible. Unfortunately it was in production so I did not have time to analyze the problem any further.
3. Directly from the netfilter code: ~src/linux/net/netfilter/nf_conntrack_core.c.
4. If I get time I will look into the netfilter tree for it's paramters.

systhread's
recommended reads

 

 

 

 

 

Submit site
news to Digg!

Slashdot It!
Bookmark on Delicious