Tag Archives: configuration

High Performance Load Balancing Using DSR on a Raspberry Pi 2

TL;DR: Direct Server Return allows a Raspberry Pi or other low-end equipment to balance load at performance impossible to achieve using other methods.

Test conditions

Pi directly connected to laptop with Ethernet.

Laptop running Debian VM in Virtual Box with adapter 1 on NAT and adapter 2 on Ethernet (bridged)

Two more VM:s in VirtualBox on the same laptop to be used as test targets, connected in the same way as the first.

One tricky aspect of this test is that DSR requires a dedicated network interface, and the Pi only has one. This means that everything needs to be set up with the interface configured normally, and then the interface must be reconfigured and the test controlled from the console.

The Debian VM is temporarily set up as gateway to the outside world.

root@debian:~# ifconfig eth1
root@debian:~# echo 1 > /proc/sys/net/ipv4/ip_forward
root@debian:~# iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

And the Pi is temporarily set up with this configuration:

Pi eth0 =
GW =

ping from pi


Install Pen

Make sure Raspbian on the Pi is up to date.

apt-get update
apt-get upgrade

Continue with instructions from the Wiki.

apt-get install automake autoconf gcc git
mkdir Git
cd Git
git clone https://github.com/UlricE/pen.git
cd pen
automake --add-missing

Verify the installation

./pen -dfU 53

And from the Debian VM:

root@debian:~# dig @ +short siag.nu

Pen is now confirmed to work.

Let’s try one more. There is an Apache server running on the Debian VM.

root@raspberrypi:~/Git/pen# ./pen -df 80

root@debian:~# lynx -dump
It works!

This is the default web page for this server.

The web server software is running but no content has been added, yet.

Configure the Pi for DSR

Everything seems good to go. We can now reconfigure eth0 on the Pi.

ifconfig eth0

./pen -df -O "dsr_if eth0" -r

This means that we intend to forward any TCP traffic with destination address to the two servers and, load balanced using round robin.

Those two addresses exist on two additional Debian VM:s on the same laptop. Like the first one they each have eth0 connected to NAT and eth1 connected to wired ethernet.

Set up test targets

Both VM:s need a loopback interface configured with the virtual address
which they must mot tell anyone about:

ifconfig lo:1
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore

Restart Apache to make sure it listens on the new address:

service apache2 restart

Finally verify that we can access Apache on all addresses:

root@debian:~# lynx -dump
It works!

This is the default web page for this server.

The web server software is running but no content has been added, yet.
root@debian:~# lynx -dump
It works!

This is the default web page for this server.

The web server software is running but no content has been added, yet.
root@debian:~# lynx -dump
It works!

This is the default web page for this server.

The web server software is running but no content has been added, yet.

And here is one of the main reasons for wanting to use DSR:

root@debian:~# ab -n 1000 -c 20
This is ApacheBench, Version 2.3 <$Revision: 1604373 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests

Server Software: Apache/2.2.22
Server Hostname:
Server Port: 80

Document Path: /1000k
Document Length: 1024000 bytes

Concurrency Level: 20
Time taken for tests: 13.427 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 1024235000 bytes
HTML transferred: 1024000000 bytes
Requests per second: 74.48 [#/sec] (mean)
Time per request: 268.532 [ms] (mean)
Time per request: 13.427 [ms] (mean, across all concurrent requests)
Transfer rate: 74496.13 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 1 7 8.3 6 239
Processing: 74 258 139.7 210 1081
Waiting: 0 11 22.4 8 239
Total: 77 265 140.0 217 1087

Percentage of the requests served within a certain time (ms)
50% 217
66% 239
75% 269
80% 390
90% 439
95% 517
98% 691
99% 782
100% 1087 (longest request)

The document 1000k is a dummy file containing 1024000 zeroes. Fetching it at 74.48 requests per second corresponds to a bandwidth of 610 Mbps, a speed physically impossible to achieve through the Pi’s Fast Ethernet interface, but easily achieved using DSR since the return traffic bypasses the load balancer completely. CPU usage on the Pi hovered at 15-20% during the test.


How to get A+ on Qualys SSL Labs Test

This requires the version of Pen currently in Git, or 0.27.4 when that is released in a few days.


For this exercise, we’ll throw compatibility with older operating systems and browsers out and only focus on maxing out security.


First, we need a 4096 bit private key. In the following, replace “your.domain” with the real domain name you’re going to protect.

openssl genrsa -out your.domain.key 4096
openssl req -sha256 -new -key your.domain.key -out your.domain.csr

Your private key is in the file your.domain.key. The file your.domain.csr contains your certificate signing request, which needs to be sent to your certification authority. The details of that procedure is different depending on the CA, but should result in you having your new certificate in your possession. Save the certificate as your.domain.crt.

The final piece of information you need is the CA’s certificate, which the CA will provide. Save the certificate as intermediate.crt.

Assuming you managed to cobble together all these files in the directory /etc/pen, the certificate installation is now finished.

Protocol Support

This is easy. Nobody supports SSL 2.0 anymore. SSL 3.0 is only for IE6 on Windows XP, a dwindling user base. TLS 1.0 is still acceptable, but this is not an exercise in acceptability (or compatibility). Throw out everything but TLS 1.2 by putting the following in /etc/pen/https.cfg:

ssl_option no_sslv2
ssl_option no_sslv3
ssl_option no_tlsv1
ssl_option no_tlsv1.1

Cipher Strength

We want ECDHE support for perfect forward secrecy, we want 256 bits encryption, and we want to prefer the best ciphers. These lines in /etc/pen/https.cfg provide that:

ssl_option cipher_server_preference

Strict Transport Security

The final piece of the puzzle is HSTS, which we accomplish by putting this in our Apache config:

Header always set Strict-Transport-Security "max-age=63072000; includeSubdomains; preload"

Finally, enable mod_headers and restart Apache:

a2enmod headers
service apache2 restart

Start Pen

The command line for Pen looks like this:

/usr/local/bin/pen -u pen -C /var/run/pen/https.ctl -F /etc/pen/https.cfg -p /var/run/pen/https.pid -K /etc/pen/your.domain.key -E /etc/pen/your.domain.crt -G /etc/pen/intermediate.crt -S 2 443

That’s quite a bit to type. If you’re using Systemd, like the CentOS system that was used for this example, here’s the full unit file to be installed into /usr/lib/systemd/system:

Description=Pen load balancer (https)
ExecStart=/usr/local/bin/pen -u pen -C /var/run/pen/https.ctl -F /etc/pen/https.cfg -p /var/run/pen/https.pid -K /etc/pen/your.domain.key -E /etc/pen/your.domain.crt -G /etc/pen/intermediate.crt -S 2 443

See this post for more on Pen and Systemd.


Pen and Systemd

Systemd is an init system for Linux, i.e. a program which runs as PID 1 and controls the startup of daemons and services. It does a bunch of other stuff as well, in a way that isn’t quite in keeping with Unix tradition, and this has caused a bit of controversy. We can ignore that for the purpose of this post.

Red Hat 7 uses systemd as its default init, as will Debian 8. Systemd isn’t configured like the familiar SysV init, so most people tasked with installing Linux servers will need to relearn. For this post, we will look at installing and configuring Pen on a CentOS 7 server.

First we need the Pen binaries. Fortunately that job has already been done for us. Pen is in the “Extra Packages for Enterprise Linux” repository, or EPEL:

yum install epel-release
yum --enablerepo=epel -y install pen

Create a user for pen to run as:

useradd pen

Create a directory for pen to keep its stuff while it is running. We can’t use /var/run because the pen user isn’t allowed to create files there, and we can’t just mkdir /var/run/pen because /var/run is a tmpfs which is recreated when the server boots. Instead we create this file in /etc/tmpfiles.d/:

# /etc/tmpfiles.d/pen.conf
d /var/run/pen 0755 pen pen -

And to actually create the directory:

systemd-tmpfiles --create

Create the configuration files, one per load balanced service. In this case, one for dns and one for http.

[root@centos7 pen]# cat /etc/pen/www.cfg
server 0 address port 80
debug 1
[root@centos7 pen]# cat /etc/pen/dns.cfg
server 0 address port 53
server 1 address port 53
debug 1

We have debugging turned on in order to get some logging. In order to get rsyslog to handle the debug messages, add this line to /etc/rsyslog.conf:

*.debug /var/log/debug

Restart rsyslog like so:

systemctl restart rsyslog

Finally add the service files that tell systemd to manage Pen.

[root@centos7 pen]# cat /usr/lib/systemd/system/pen-www.service
Description=Pen load balancer (www)
ExecStart=/usr/bin/pen -u pen -C /var/run/pen/www.ctl -F /etc/pen/www.cfg -p /var/run/pen/www.pid -S 2 80
[root@centos7 pen]# cat /usr/lib/systemd/system/pen-dns.service
Description=pen load balancer (dns)
ExecStart=/usr/bin/pen -u pen -C /var/run/pen/dns.ctl -F /etc/pen/dns.cfg -p /var/run/pen/dns.pid -S 2 -U :::53

Make systemd reread its configuration. That can actually be done in the oldfashioned way:

kill -HUP 1

Now we can use systemctl to start and stop the services:

[root@centos7 pen]# systemctl start pen-www
[root@centos7 pen]# systemctl start pen-dns
[root@centos7 pen]# systemctl status pen-dns
pen-dns.service - pen load balancer (dns)
Loaded: loaded (/usr/lib/systemd/system/pen-dns.service; static)
Active: active (running) since Sun 2014-10-19 02:53:36 CEST; 38s ago
Process: 19004 ExecStart=/usr/bin/pen -u pen -C /var/run/pen/dns.ctl -F /etc/pen/dns.cfg -p /var/run/pen/dns.pid -S 2 -U :::53 (code=exited, status=0/SUCCESS)
Main PID: 19005 (pen)
CGroup: /system.slice/pen-dns.service
└─19005 /usr/bin/pen -u pen -C /var/run/pen/dns.ctl -F /etc/pen/dns.cfg -p /var/run/pen/dns.pid -S 2 -U :::53

Oct 19 02:53:36 centos7 systemd[1]: Starting pen load balancer (dns)...
Oct 19 02:53:36 centos7 systemd[1]: PID file /var/run/pen/dns.pid not readable (yet?) after start.
Oct 19 02:53:36 centos7 systemd[1]: Started pen load balancer (dns).
[root@centos7 pen]# systemctl status pen-www
pen-www.service - Pen load balancer (www)
Loaded: loaded (/usr/lib/systemd/system/pen-www.service; static)
Active: active (running) since Sun 2014-10-19 02:53:33 CEST; 47s ago
Process: 19000 ExecStart=/usr/bin/pen -u pen -C /var/run/pen/www.ctl -F /etc/pen/www.cfg -p /var/run/pen/www.pid -S 2 80 (code=exited, status=0/SUCCESS)
Main PID: 19001 (pen)
CGroup: /system.slice/pen-www.service
└─19001 /usr/bin/pen -u pen -C /var/run/pen/www.ctl -F /etc/pen/www.cfg -p /var/run/pen/www.pid -S 2 80

Oct 19 02:53:33 centos7 systemd[1]: Starting Pen load balancer (www)...
Oct 19 02:53:33 centos7 systemd[1]: PID file /var/run/pen/www.pid not readable (yet?) after start.
Oct 19 02:53:33 centos7 systemd[1]: Started Pen load balancer (www).

Let’s check that it works, too.

[root@centos7 pen]# dig +short @localhost siag.nu

What a relief. 🙂

Thanks to Djamel Ouerdi for helping me figure out how this works.


Security and Pen

Prompted by this:


Summary: use non-default features but none of the security ones in Pen and you can end up with something not very secure. First I thought “why do that?” but realized that Debian ship Pen without a lot of configuration hints. So here are a few:

  • Don’t run Pen as root
  • Use a jail
  • Use access lists to limit access


Here’s what needs to be done to create a chroot jail for Pen and run it there as a non-root user. Start/stop script added.

useradd pen
mkdir -p /var/lib/pen/etc /var/lib/pen/tmp
chown pen /var/lib/pen/tmp
grep ^pen: /etc/passwd > /var/lib/pen/etc/passwd
cat << EOF > /var/lib/pen/etc/pen.cfg
acl 0 deny
control_acl 0
acl 1 deny
client_acl 1
server 0 address port 88
cat << EOF > /etc/init.d/pen

case "\$1" in
start )
        pen -j /var/lib/pen -C 10080 -F /etc/pen.cfg -p /tmp/pen.pid 8080
stop )
        kill \`cat /var/lib/pen/tmp/pen.pid\`
* )
        echo "Usage: \$0 start|stop"
chmod +x /etc/init.d/pen