Tag Archives: Linux

Tarpit support in Pen

Pen 0.32 will have built-in tarpit support in its Direct Server Return mode. The feature is enabled by specifying an access control list against which incoming requests are matched. Matching destination addresses will make Pen do two things:

1. Reply to ARP requests to such addresses.
2. Reply to TCP SYN with SYN+ACK.

The idea behind tarpitting is to slow down network scanning by giving lots of false positives. Pen does this with very little load and without having to manage any state.

Here is an example command line.

pen -df -O “acl 1 permit” -O “tarpit_acl 1” -O “dsr_if eth1”

Let’s go through that option by option.

acl 1 permit creates an entry in access list 1 which matches IP address All other IP addresses will be rejected.

tarpit_acl 1 makes Pen use access list 1, the one with as its sole entry, to match destination addresses.

dsr_if eth1 makes Pen use eth1 as the network interface where all direct server return processing is performed. is the address and port where Pen listens for legitimate requests. They will be forwarded to the backend servers. and are the backend servers. They have web servers listening on port 80 and IP address configured on a loopback interface. See the Wiki.

Let’s try making a legitimate request.

ulric@debtest:~/Git/pen$ curl

That worked fine. Frames from us go to Pen, Pen forwards them to one of the web servers, the web server replies directly to us. In Wireshark we see:


But what happens when we try the same thing on a tarpitted address?

ulric@debtest:~/Git/pen$ curl

It just hangs. We send SYN, Pen replies with SYN+ACK, we send ACK and think that the TCP handshake is done. So we send the HTTP request, which Pen ignores. We send it again. Pen ignores it again, and so on. Here’s what that looks like in Wireshark:


Access control lists are a very flexible way to control the tarpit functionality in Pen and have it tarpit every address in a subnet (except those that shouldn’t). As an example, think of a network with the following hosts: gateway web server 1 web server 2 load balanced address

The corresponding ACL would be created like this:

acl 1 deny
acl 1 deny
acl 1 deny
acl 1 deny

Anything Pen sees that is not destined for one of these addresses will be tarpitted.


Direct Server Return for UDP

Pen has supported Direct Server Return for TCP for some time. Support for UDP has now been added, suitable for load balancing e.g. DNS.

Here, debian2 is the DNS client and debian uses Pen in DSR mode to load balance between debian3 and debian4 running Bind.

Pen command line:

ulric@debian:~/Git/pen$ sudo ./pen -df -U -O poll -O “dsr_if eth1” -S 2 -r
As of 0.28.1 the server table is expanded dynamically,
making the -S option obsolete
2015-08-03 16:24:09: read_cfg((null))
2015-08-03 16:24:09: Before: conns = (nil), connections_max = 0, clients = (nil), clients_max = 0
2015-08-03 16:24:09: expand_conntable(500)
2015-08-03 16:24:09: After: conns = 0x1ac4600, connections_max = 500, clients = 0x7f6428d5c010, clients_max = 2048
2015-08-03 16:24:09: pen 0.29.0 starting
2015-08-03 16:24:09: servers:
2015-08-03 16:24:09: 0
2015-08-03 16:24:09: 1

As far as debian2 can see, the responses are coming from a single DNS server:


But tcpdump on debian3 and debian4 shows requests and replies being load balanced across the hosts:




Pen 0.29.0 released

Available here:


And also here:


Pen 0.29.0 introduces transparent reverse proxying on supported platforms,
which currently means Linux, FreeBSD and OpenBSD. This allows the backend
servers to see the client’s real address. It can be used in combination
with SSL termination.

Another improvement is that the server table size is no longer fixed
at startup but grows dynamically as servers are added. The -S option is
still accepted but doesn’t do anything. The client and connection tables
can also be expanded on the fly, reducing the number of restarts.

Full list of changes from 0.28.0:

150608 Released 0.29.0.

150528 Transparent reverse proxy support for Linux, FreeBSD and OpenBSD.

150527 Allow the client table size to be updated on the fly. Default size still 2048.
Allow the connection table size to be updated in the fly. Default still 500.
See penctl.1, options clients_max and conn_max.

150526 Introduced the macro NO_SERVER to be used instead of -1 to signify
error conditions and such.
Removed the fixed server table size along with the -S option.

150525 Fixed cosmetic bug in startup code which required port to be specified
on backend servers even if it was the same as the listening port.


Pen 0.28.0 released

Available here:


And also here:


Pen 0.28.0 brings Direct Server Return on Linux and FreeBSD.

It also brings the Windows code up to speed.

Full list of changes from 0.27.5:

150520 Released 0.28.0.

150513 Numerous updates to support the madness that is Windows.

150501 Fix from Vincent Bernat: segfault when not using SSL.

150427 DSR support using Netmap on FreeBSD.
Unbroke DSR on Linux.

150424 Replaced all calls to perror with debug(…, strerror(errno);
Updated penlog and penlogd to use diag.[ch].

150422 More refactoring: broke out conn.[ch], client.[ch], server.[ch],
Made a hash index such that the load balancer may balance load.

150420 Broke out Windows code from pen.c into windows.c. Added windows.h.

150419 Broke out public definitions for dsr into dsr.h.
Broke out memory management into memory.[ch].
Broke out dignostic and logging functions into diag.[ch].
Broke out settings into settings.[ch].
Broke out access lists into acl.[ch].
Broke out event initialization into event.[ch].
Added pen_epoll.h, pen_kqueue.h, pen_poll.h, pen_select.h.
Broke out pen_aton et al into netconv.[ch].

150416 Added dsr.c


Comparing Raw Networking on Linux and FreeBSD

Direct Server Return requires raw access to network traffic. There is no portable way to do that, so Pen supports it so far only on Linux and FreeBSD.

On Linux, the probe into the network stack is created pretty much like any other socket:


It works literally like a probe, the packets are duplicated out and received both by the kernel and by the program creating the socket, Pen in this case. Care must be taken to prevent the kernel from interferring, for example by responding to TCP traffic.

Reading and writing is as usual:

recvfrom(fd, buf, MAXBUF, 0, NULL, NULL);
sendto(fd, b, len, 0, NULL, 0);

FreeBSD has a totally different solution called Netmap:

d = nm_open(ifname, NULL, 0, 0);

Here, d is not a socket but a “netmap descriptor” and reading and writing is done to rings of buffers, matching what is available in the network card. The regular network stack is cut off from the traffic. The details are hidden behind a pcap-like api:

nm_nextpkt(d, &h);
nm_inject(d, b, len);

Now, wouldn’t it be fun to compare these two? Of course it would!

Two VMs are prepared with Apache and the address on a loopback interface.

One VM runs ApacheBench like this:

ab -c 20 -n 10000

One Linux and one FreeBSD VM are prepared with the latest Pen from Git. On Linux the command line looks like:

sudo ./pen -df -O poll -O "dsr_if eth1" -S 2 -r

And on FreeBSD:

sudo ./pen -df -O poll -O "dsr_if em1" -S 2 -r

I.e. exactly the same, only the interface name differs.

Linux results here, ~0.9 Gbps:


And FreeBSD results, ~1.4 Gbps: