Tag Archives: performance

Comparing Raw Networking on Linux and FreeBSD

Direct Server Return requires raw access to network traffic. There is no portable way to do that, so Pen supports it so far only on Linux and FreeBSD.

On Linux, the probe into the network stack is created pretty much like any other socket:


It works literally like a probe, the packets are duplicated out and received both by the kernel and by the program creating the socket, Pen in this case. Care must be taken to prevent the kernel from interferring, for example by responding to TCP traffic.

Reading and writing is as usual:

recvfrom(fd, buf, MAXBUF, 0, NULL, NULL);
sendto(fd, b, len, 0, NULL, 0);

FreeBSD has a totally different solution called Netmap:

d = nm_open(ifname, NULL, 0, 0);

Here, d is not a socket but a “netmap descriptor” and reading and writing is done to rings of buffers, matching what is available in the network card. The regular network stack is cut off from the traffic. The details are hidden behind a pcap-like api:

nm_nextpkt(d, &h);
nm_inject(d, b, len);

Now, wouldn’t it be fun to compare these two? Of course it would!

Two VMs are prepared with Apache and the address on a loopback interface.

One VM runs ApacheBench like this:

ab -c 20 -n 10000

One Linux and one FreeBSD VM are prepared with the latest Pen from Git. On Linux the command line looks like:

sudo ./pen -df -O poll -O "dsr_if eth1" -S 2 -r

And on FreeBSD:

sudo ./pen -df -O poll -O "dsr_if em1" -S 2 -r

I.e. exactly the same, only the interface name differs.

Linux results here, ~0.9 Gbps:


And FreeBSD results, ~1.4 Gbps:



The Great Load Balancer Shootout…

… is not what this is about. I was however curious how Pen performs compared to other open source load balancers. All load balancers I could think of in Raspbian were installed on the trusty RPi2 and configured to withstand the same torrent of traffic I use to test Pen.

The result can be seen below. In the interest of keeping crazies out, the “other” load balancers have been anonymized. This is a very simple synthetic test and the results may vary a lot depending on the test conditions. It is however nice to see that:

– Pen had the highest maximum throughput, measured in requests per second
– Pen supported the highest concurrency of all tested load balancers

In the chart below, X is the number of concurrent requests and Y is the number of requests per second. More is better. No line means that the load balancer failed.