Tag Archives: Direct Server Return

Pen 0.32.0 released

Available here:


And also here:


Pen 0.32.0 adds tarpit functionality to the Direct Server Return mode. The purpose of tarpitting is to make network scanning harder by producing lots of false positives.

Full list of changes from 0.31.1:

151123 Released 0.32.0.

151120 Added tarpit test case to testsuite.sh.

151117 Tarpit functionality to be used with the DSR mode.

151112 pen.1: removed obsolete -S option, updated defaults for -x and -L.


Tarpit support in Pen

Pen 0.32 will have built-in tarpit support in its Direct Server Return mode. The feature is enabled by specifying an access control list against which incoming requests are matched. Matching destination addresses will make Pen do two things:

1. Reply to ARP requests to such addresses.
2. Reply to TCP SYN with SYN+ACK.

The idea behind tarpitting is to slow down network scanning by giving lots of false positives. Pen does this with very little load and without having to manage any state.

Here is an example command line.

pen -df -O “acl 1 permit” -O “tarpit_acl 1” -O “dsr_if eth1”

Let’s go through that option by option.

acl 1 permit creates an entry in access list 1 which matches IP address All other IP addresses will be rejected.

tarpit_acl 1 makes Pen use access list 1, the one with as its sole entry, to match destination addresses.

dsr_if eth1 makes Pen use eth1 as the network interface where all direct server return processing is performed. is the address and port where Pen listens for legitimate requests. They will be forwarded to the backend servers. and are the backend servers. They have web servers listening on port 80 and IP address configured on a loopback interface. See the Wiki.

Let’s try making a legitimate request.

ulric@debtest:~/Git/pen$ curl

That worked fine. Frames from us go to Pen, Pen forwards them to one of the web servers, the web server replies directly to us. In Wireshark we see:


But what happens when we try the same thing on a tarpitted address?

ulric@debtest:~/Git/pen$ curl

It just hangs. We send SYN, Pen replies with SYN+ACK, we send ACK and think that the TCP handshake is done. So we send the HTTP request, which Pen ignores. We send it again. Pen ignores it again, and so on. Here’s what that looks like in Wireshark:


Access control lists are a very flexible way to control the tarpit functionality in Pen and have it tarpit every address in a subnet (except those that shouldn’t). As an example, think of a network with the following hosts: gateway web server 1 web server 2 load balanced address

The corresponding ACL would be created like this:

acl 1 deny
acl 1 deny
acl 1 deny
acl 1 deny

Anything Pen sees that is not destined for one of these addresses will be tarpitted.


High Performance Load Balancing Using DSR on a Raspberry Pi 2

TL;DR: Direct Server Return allows a Raspberry Pi or other low-end equipment to balance load at performance impossible to achieve using other methods.

Test conditions

Pi directly connected to laptop with Ethernet.

Laptop running Debian VM in Virtual Box with adapter 1 on NAT and adapter 2 on Ethernet (bridged)

Two more VM:s in VirtualBox on the same laptop to be used as test targets, connected in the same way as the first.

One tricky aspect of this test is that DSR requires a dedicated network interface, and the Pi only has one. This means that everything needs to be set up with the interface configured normally, and then the interface must be reconfigured and the test controlled from the console.

The Debian VM is temporarily set up as gateway to the outside world.

root@debian:~# ifconfig eth1
root@debian:~# echo 1 > /proc/sys/net/ipv4/ip_forward
root@debian:~# iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

And the Pi is temporarily set up with this configuration:

Pi eth0 =
GW =

ping from pi


Install Pen

Make sure Raspbian on the Pi is up to date.

apt-get update
apt-get upgrade

Continue with instructions from the Wiki.

apt-get install automake autoconf gcc git
mkdir Git
cd Git
git clone https://github.com/UlricE/pen.git
cd pen
automake --add-missing

Verify the installation

./pen -dfU 53

And from the Debian VM:

root@debian:~# dig @ +short siag.nu

Pen is now confirmed to work.

Let’s try one more. There is an Apache server running on the Debian VM.

root@raspberrypi:~/Git/pen# ./pen -df 80

root@debian:~# lynx -dump
It works!

This is the default web page for this server.

The web server software is running but no content has been added, yet.

Configure the Pi for DSR

Everything seems good to go. We can now reconfigure eth0 on the Pi.

ifconfig eth0

./pen -df -O "dsr_if eth0" -r

This means that we intend to forward any TCP traffic with destination address to the two servers and, load balanced using round robin.

Those two addresses exist on two additional Debian VM:s on the same laptop. Like the first one they each have eth0 connected to NAT and eth1 connected to wired ethernet.

Set up test targets

Both VM:s need a loopback interface configured with the virtual address
which they must mot tell anyone about:

ifconfig lo:1
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore

Restart Apache to make sure it listens on the new address:

service apache2 restart

Finally verify that we can access Apache on all addresses:

root@debian:~# lynx -dump
It works!

This is the default web page for this server.

The web server software is running but no content has been added, yet.
root@debian:~# lynx -dump
It works!

This is the default web page for this server.

The web server software is running but no content has been added, yet.
root@debian:~# lynx -dump
It works!

This is the default web page for this server.

The web server software is running but no content has been added, yet.

And here is one of the main reasons for wanting to use DSR:

root@debian:~# ab -n 1000 -c 20
This is ApacheBench, Version 2.3 <$Revision: 1604373 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests

Server Software: Apache/2.2.22
Server Hostname:
Server Port: 80

Document Path: /1000k
Document Length: 1024000 bytes

Concurrency Level: 20
Time taken for tests: 13.427 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 1024235000 bytes
HTML transferred: 1024000000 bytes
Requests per second: 74.48 [#/sec] (mean)
Time per request: 268.532 [ms] (mean)
Time per request: 13.427 [ms] (mean, across all concurrent requests)
Transfer rate: 74496.13 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 1 7 8.3 6 239
Processing: 74 258 139.7 210 1081
Waiting: 0 11 22.4 8 239
Total: 77 265 140.0 217 1087

Percentage of the requests served within a certain time (ms)
50% 217
66% 239
75% 269
80% 390
90% 439
95% 517
98% 691
99% 782
100% 1087 (longest request)

The document 1000k is a dummy file containing 1024000 zeroes. Fetching it at 74.48 requests per second corresponds to a bandwidth of 610 Mbps, a speed physically impossible to achieve through the Pi’s Fast Ethernet interface, but easily achieved using DSR since the return traffic bypasses the load balancer completely. CPU usage on the Pi hovered at 15-20% during the test.


Direct Server Return for UDP

Pen has supported Direct Server Return for TCP for some time. Support for UDP has now been added, suitable for load balancing e.g. DNS.

Here, debian2 is the DNS client and debian uses Pen in DSR mode to load balance between debian3 and debian4 running Bind.

Pen command line:

ulric@debian:~/Git/pen$ sudo ./pen -df -U -O poll -O “dsr_if eth1” -S 2 -r
As of 0.28.1 the server table is expanded dynamically,
making the -S option obsolete
2015-08-03 16:24:09: read_cfg((null))
2015-08-03 16:24:09: Before: conns = (nil), connections_max = 0, clients = (nil), clients_max = 0
2015-08-03 16:24:09: expand_conntable(500)
2015-08-03 16:24:09: After: conns = 0x1ac4600, connections_max = 500, clients = 0x7f6428d5c010, clients_max = 2048
2015-08-03 16:24:09: pen 0.29.0 starting
2015-08-03 16:24:09: servers:
2015-08-03 16:24:09: 0
2015-08-03 16:24:09: 1

As far as debian2 can see, the responses are coming from a single DNS server:


But tcpdump on debian3 and debian4 shows requests and replies being load balanced across the hosts:




Pen 0.28.0 released

Available here:


And also here:


Pen 0.28.0 brings Direct Server Return on Linux and FreeBSD.

It also brings the Windows code up to speed.

Full list of changes from 0.27.5:

150520 Released 0.28.0.

150513 Numerous updates to support the madness that is Windows.

150501 Fix from Vincent Bernat: segfault when not using SSL.

150427 DSR support using Netmap on FreeBSD.
Unbroke DSR on Linux.

150424 Replaced all calls to perror with debug(…, strerror(errno);
Updated penlog and penlogd to use diag.[ch].

150422 More refactoring: broke out conn.[ch], client.[ch], server.[ch],
Made a hash index such that the load balancer may balance load.

150420 Broke out Windows code from pen.c into windows.c. Added windows.h.

150419 Broke out public definitions for dsr into dsr.h.
Broke out memory management into memory.[ch].
Broke out dignostic and logging functions into diag.[ch].
Broke out settings into settings.[ch].
Broke out access lists into acl.[ch].
Broke out event initialization into event.[ch].
Added pen_epoll.h, pen_kqueue.h, pen_poll.h, pen_select.h.
Broke out pen_aton et al into netconv.[ch].

150416 Added dsr.c


Comparing Raw Networking on Linux and FreeBSD

Direct Server Return requires raw access to network traffic. There is no portable way to do that, so Pen supports it so far only on Linux and FreeBSD.

On Linux, the probe into the network stack is created pretty much like any other socket:


It works literally like a probe, the packets are duplicated out and received both by the kernel and by the program creating the socket, Pen in this case. Care must be taken to prevent the kernel from interferring, for example by responding to TCP traffic.

Reading and writing is as usual:

recvfrom(fd, buf, MAXBUF, 0, NULL, NULL);
sendto(fd, b, len, 0, NULL, 0);

FreeBSD has a totally different solution called Netmap:

d = nm_open(ifname, NULL, 0, 0);

Here, d is not a socket but a “netmap descriptor” and reading and writing is done to rings of buffers, matching what is available in the network card. The regular network stack is cut off from the traffic. The details are hidden behind a pcap-like api:

nm_nextpkt(d, &h);
nm_inject(d, b, len);

Now, wouldn’t it be fun to compare these two? Of course it would!

Two VMs are prepared with Apache and the address on a loopback interface.

One VM runs ApacheBench like this:

ab -c 20 -n 10000

One Linux and one FreeBSD VM are prepared with the latest Pen from Git. On Linux the command line looks like:

sudo ./pen -df -O poll -O "dsr_if eth1" -S 2 -r

And on FreeBSD:

sudo ./pen -df -O poll -O "dsr_if em1" -S 2 -r

I.e. exactly the same, only the interface name differs.

Linux results here, ~0.9 Gbps:


And FreeBSD results, ~1.4 Gbps:



Direct Server Return

Direct server return means that traffic from the client to the server goes through Pen, but the traffic from the server to the client does not. There are a number of advantages to doing it this way, especially when it comes to performance – the load balancer never has to touch the return traffic. There are disadvantages as well, for example the load balancer cannot do SSL termination and the servers need special configuration.

Very, very early development, but it already kind of works. Ignore the mediocre performance, the code is running with debug level “blazing madness”.