Comparing Raw Networking on Linux and FreeBSD

Direct Server Return requires raw access to network traffic. There is no portable way to do that, so Pen supports it so far only on Linux and FreeBSD.

On Linux, the probe into the network stack is created pretty much like any other socket:


It works literally like a probe, the packets are duplicated out and received both by the kernel and by the program creating the socket, Pen in this case. Care must be taken to prevent the kernel from interferring, for example by responding to TCP traffic.

Reading and writing is as usual:

recvfrom(fd, buf, MAXBUF, 0, NULL, NULL);
sendto(fd, b, len, 0, NULL, 0);

FreeBSD has a totally different solution called Netmap:

d = nm_open(ifname, NULL, 0, 0);

Here, d is not a socket but a “netmap descriptor” and reading and writing is done to rings of buffers, matching what is available in the network card. The regular network stack is cut off from the traffic. The details are hidden behind a pcap-like api:

nm_nextpkt(d, &h);
nm_inject(d, b, len);

Now, wouldn’t it be fun to compare these two? Of course it would!

Two VMs are prepared with Apache and the address on a loopback interface.

One VM runs ApacheBench like this:

ab -c 20 -n 10000

One Linux and one FreeBSD VM are prepared with the latest Pen from Git. On Linux the command line looks like:

sudo ./pen -df -O poll -O "dsr_if eth1" -S 2 -r

And on FreeBSD:

sudo ./pen -df -O poll -O "dsr_if em1" -S 2 -r

I.e. exactly the same, only the interface name differs.

Linux results here, ~0.9 Gbps:


And FreeBSD results, ~1.4 Gbps:



Interesting observation regarding virtio-net

Interesting observation made while testing direct server return through Pen in VirtualBox.

Using e1000 drivers will send 1514 byte frames. Pen receives the frames, scribbles a bit on the headers and puts them back on the network. Iperf performance through Pen is pretty decent, 200 Mbps. Direct iperf without Pen in between is 400 Mbps. That makes sense, total throughput is limited by the host running all three virtual servers.

With virtio drivers, iperf performance without Pen is 1 Gbps. But performance with Pen drops to < 100 kbps! What's up with that? It turns out that the virtio nic drivers support TCP Segmentation Offload and will send oversized frames, expecting the physical nic to do the frame segmentation. With an entirely virtual network, no segmentation takes place. As a result, Pen receives oversized frames which it can't forward. The frames are dropped and TCP gets grumpy. To get rid of this behaviour, TSO must be turned off on the iperf client: ethtool -K eth1 tso off

However, this makes direct iperf performance between two vm:s drop from 1 Gbps to 365 Mbps. Also, iperf performance through Pen drops from 200 Mbps to 60 Mbps. So virtio nics are actually slower for this purpose than software emulated e1000.


Direct Server Return

Direct server return means that traffic from the client to the server goes through Pen, but the traffic from the server to the client does not. There are a number of advantages to doing it this way, especially when it comes to performance – the load balancer never has to touch the return traffic. There are disadvantages as well, for example the load balancer cannot do SSL termination and the servers need special configuration.

Very, very early development, but it already kind of works. Ignore the mediocre performance, the code is running with debug level “blazing madness”.




Pen 0.27.4 released

Available here:

And also here:

Pen 0.27.4 rounds off the SSL support with several TLS extensions and security related enhancements.

TLS 1.1 and TLS 1.2 can now be individually disabled.

Client-initiated secure renegotiation can now be rate-limited. The default “limit” is once per hour, which effectively disables this potential vector for denial of service.

OCSP stapling is now supported.

Server Name Indication is supported.

See penctl manpage for configuration help.

Full list of changes from 0.27.3:

150408 Updated ocsp stapling to be compatible with server name indication.
Added script.
Released 0.27.4.

150407 SSL code broken out into ssl.[ch]. SSL context creation broken
out from ssl_init to ssl_create_context.
Server Name Indication support. New command to enable:
ssl_sni_path PATH
where PATH is the name of a directory containing domain.key,
domain.crt and files for each domain.

150406 OCSP stapling. New command ssl_ocsp_response filename
specifies the location of the ocsp response to be stapled.
The response must be pre-fetched. The idea was borrowed
from Rob Stradling.

150403 New command ssl_client_renegotiation_interval specifies the
minimum number of seconds the client must wait between
renegotiation requests. Default 3600.

150402 Enabled SSL session resumption.
In do_cmd: don’t print “ignoring command” for comments starting
with ‘#’.

150330 Added ssl_option no_tlsv1.1 and ssl_option no_tlsv1.2 to disable
SSL 1.1 and 1.2 respectively.


Pen and Server Name Indication

Server Name Indication is a TLS extension which allows multiple virtual hosts, and multiple digital certificates, to coexist on the same IP address and port. It is among the features destined for Pen 0.27.4.

To enable SNI in Pen, all that needs to be done is this line in the configuration file:

ssl_sni_path /etc/pen/sni

Then add the private key, your certificate and the CA’s certificate to that directory, using this naming scheme:

[root@lb pen]# ls /etc/pen/sni

Now any time a request comes in for, these files will be used for the negotiation.

The downside is that a reasonably modern browser is required.



OCSP Stapling Support in Pen

Pen 0.27.4 introduces support for OCSP stapling. What the heck is this, you may ask.

Https websites use certificates to prove that they are who they claim they are. The certificates are signed by trusted entities, called certificate authorities. Trusted by whom? The person using the browser most likely have no idea what the CA:s are or who works there. But the browser has a built-in list of CA:s that it trusts. It is at least a little bit of trust, so let’s say that a little trust is better than no trust.

The certificates are good for a year or more, after which time they need to be renewed. But what if the certificate needs to be revoked before its time is up?

Kludge upon kludge upon kludge

One such scheme is OCSP, “online certificate status protocol”, where the browser connects to the CA and asks whether the certificate is still valid. Unlike the certificate itself, which has a life span of years, the OCSP response has a much shorter life of hours.

One may think that if the certificate had a shorter life to begin with, none of this would be necessary. But let’s ignore that for now. Surely this fixes the problem with trusting certificates that shouldn’t be trusted?

Well, not quite. There are a bunch of problems with OCSP. For one, it requires that the browser is free to connect to the OCSP responder (the thingy on the CA:s web site that says the certificate is good). If the guys administrating the network haven’t opened the firewall to every potential OCSP responder (remember that list of “trusted” certificate authorities in the browser), OCSP doesn’t work.

By the way, care to guess what a browser does by default if the certificate validity can’t be verified? It doesn’t care. It just carries on as if the certificate were good.

There are more problems with OCSP: it creates a lot of traffic to the CA:s, traffic that they have to pay for, making operating a CA more expensive. And the queries disclose to the CA what web sites the user visits. And the queries are typically unencrypted, so anyone with a wiretap knows too. Let’s make a list.

– The user visiting knows (we hope)
– Shady Business knows
– Shady Business’ CA knows
– Every governmental agency on the way from Shady Business to the CA knows

This is obviously totally broken. It is less secure than if Shady Business had used a self-signed certificate. It is less secure than if Shady Business had used plain http.

Pull out the Swingline

What if the web server, instead of relying on the browser connecting to the CA to check the certificate status, sent the OCSP response itself? The response would still need to be pulled from the CA, but by the certificate owner, and only every few hours. This solves the traffic problem for the CA, it solves the privacy issue, and it saves the networking guys a bit of firewall configuration.

So how does one implement this? Let’s say, in an SSL-terminating load balancer?

A clue

Rob Stradling proposed this approach for Nginx:

You’ve got me thinking…

In the Apache httpd implementation, mod_ssl contains code which regularly
downloads and caches OCSP Responses in shared memory so that they’re ready to
be Stapled.

It occurs to me that the same result could be achieved without having to put
the downloading/caching code inside the webserver. A cronjob could run a
shell script that…
– runs the OpenSSL command-line tool to download the appropriate OCSP
Responses, saving them to the hard disk.
– signals the webserver to reload its config file.

With this cronjob-based approach, the webserver would only require a small
amount of new code:
– a new config directive for specifying the filename of an OCSP Response (or
perhaps the code could just look for .ors: if present, use
it for Stapling; if absent, Stapling is disabled).
– code to read the OCSP Response file.
– a call to SSL_CTX_set_tlsext_status_cb().
– some sanity checks on the OCSP Response (e.g. is it still time-valid?)
– a call to SSL_set_tlsext_status_ocsp_resp() to actually staple the OCSP

This cronjob-based approach should perform better than Apache/mod_ssl’s
approach, because it would avoid the use of a mutexed shared-memory cache.

Eminently, commendably simple. Let’s do it.


ulric@debian:~/Git/pen$ man SSL_CTX_set_tlsext_status_cb
No manual entry for SSL_CTX_set_tlsext_status_cb
ulric@debian:~/Git/pen$ man SSL_set_tlsext_status_ocsp_resp
No manual entry for SSL_set_tlsext_status_ocsp_resp

This is actually the typical situation when you try to do anything with OpenSSL. The manpages, when they exist, document details but not the big picture, and when they don’t exist they don’t document anything.


After an actually very small amount of trial and error, here is a simplified version of the callback that does the stapling:

static int ssl_stapling_cb(SSL *ssl, void *p)
unsigned char *ocsp_resp_copy;

if (ocsp_resp_file) {
int f = open(ocsp_resp_file, O_RDONLY);
ocsp_resp_len = read(f, ocsp_resp_data, OCSP_RESP_MAX);
ocsp_resp_file = NULL;
ocsp_resp_copy = pen_malloc(ocsp_resp_len);
memcpy(ocsp_resp_copy, ocsp_resp_data, ocsp_resp_len);
SSL_set_tlsext_status_ocsp_resp(ssl, ocsp_resp_copy, ocsp_resp_len);

The real code also has error checking.

Putting it together

We now have a way to use the OCSP response, once it has found its way into a file. But how did it get there?

First we need to find out what the OCSP responder URI is. Fortunately that information is embedded in the certificate itself:

openssl x509 -noout -ocsp_uri -in /etc/pen/

And to get that into the file:

openssl ocsp -noverify -issuer /etc/pen/globalsign-intermediate.crt -cert /etc/pen/ -url -header Host -respout /etc/pen/

Finally, to get pen to reload the file (it has a short life span, remember):

penctl /var/run/pen/https.ctl "ssl_ocsp_response /etc/pen/"


How to get A+ on Qualys SSL Labs Test

This requires the version of Pen currently in Git, or 0.27.4 when that is released in a few days.


For this exercise, we’ll throw compatibility with older operating systems and browsers out and only focus on maxing out security.


First, we need a 4096 bit private key. In the following, replace “your.domain” with the real domain name you’re going to protect.

openssl genrsa -out your.domain.key 4096
openssl req -sha256 -new -key your.domain.key -out your.domain.csr

Your private key is in the file your.domain.key. The file your.domain.csr contains your certificate signing request, which needs to be sent to your certification authority. The details of that procedure is different depending on the CA, but should result in you having your new certificate in your possession. Save the certificate as your.domain.crt.

The final piece of information you need is the CA’s certificate, which the CA will provide. Save the certificate as intermediate.crt.

Assuming you managed to cobble together all these files in the directory /etc/pen, the certificate installation is now finished.

Protocol Support

This is easy. Nobody supports SSL 2.0 anymore. SSL 3.0 is only for IE6 on Windows XP, a dwindling user base. TLS 1.0 is still acceptable, but this is not an exercise in acceptability (or compatibility). Throw out everything but TLS 1.2 by putting the following in /etc/pen/https.cfg:

ssl_option no_sslv2
ssl_option no_sslv3
ssl_option no_tlsv1
ssl_option no_tlsv1.1

Cipher Strength

We want ECDHE support for perfect forward secrecy, we want 256 bits encryption, and we want to prefer the best ciphers. These lines in /etc/pen/https.cfg provide that:

ssl_option cipher_server_preference

Strict Transport Security

The final piece of the puzzle is HSTS, which we accomplish by putting this in our Apache config:

Header always set Strict-Transport-Security "max-age=63072000; includeSubdomains; preload"

Finally, enable mod_headers and restart Apache:

a2enmod headers
service apache2 restart

Start Pen

The command line for Pen looks like this:

/usr/local/bin/pen -u pen -C /var/run/pen/https.ctl -F /etc/pen/https.cfg -p /var/run/pen/ -K /etc/pen/your.domain.key -E /etc/pen/your.domain.crt -G /etc/pen/intermediate.crt -S 2 443

That’s quite a bit to type. If you’re using Systemd, like the CentOS system that was used for this example, here’s the full unit file to be installed into /usr/lib/systemd/system:

Description=Pen load balancer (https)
ExecStart=/usr/local/bin/pen -u pen -C /var/run/pen/https.ctl -F /etc/pen/https.cfg -p /var/run/pen/ -K /etc/pen/your.domain.key -E /etc/pen/your.domain.crt -G /etc/pen/intermediate.crt -S 2 443

See this post for more on Pen and Systemd.


Pen 0.27.3 released

Available here:

And also here:

Several new configuration options dealing with securing SSL.

ssl_option no_sslv2 turns off SSL2. This has been the default for ages. Nobody should use SSL2 anymore.

ssl_option no_sslv3 turns off SSL3, sacrificing compatibility with Windows XP but also “sacrificing” the associated vulnerabilities.

ssl_option no_tlsv1 turns off TLS1, again sacrificing a bit of compatibility for a bit of security.

ssl_option cipher_server_preference Prefer the ciphers listed at the beginning of the cipher list (see next item).

ssl_ciphers CIPHERS Specify a list of ciphers to support. By default, Pen will use whatever OpenSSL thinks the default should be, and that list will be different depending on the version of OpenSSL and the options used when compiling OpenSSL.

See here for a suggested configuration with intermediate compatibility but still good security:

Perfect Forward Secrecy

The default maximum number of connections has been 256 since Pen’s inception in 2000. Today that is ridiculously conservative since Pen will gladly handle tens of thousands of connections on a Raspberry Pi:

The Great Load Balancer Shootout…
Let’s double that one more time

The default is now bumped to 500; still very conservative.

Full list of changes since 0.27.2:

150330 Added autoconf check that the ECDHE is available and not disabled.
Bumped default max connections and listen queue to 500.

150326 Support for ECDHE cipher suites.

150325 New commands ssl_option and ssl_ciphers to individually disable
insecure protocols and ciphers.

150324 Updated penctl.1 with the new command.

150322 New knob to tweak max number of pending nonblocking connection
attempts: pending_max N (default 100).


Perfect Forward Secrecy

One of the new features in 0.27.3 will be perfect forward secrecy for SSL.–but-no-one-uses-it.html

Enabling perfect forward secrecy involves picking an up to date version of OpenSSL compiled with the right options and using the appropriate ciphersuite. Do note that older clients (Windows XP) are at odds with secure SSL configuration – there’s no way to get both right at the same time. This suggested configuration is a compromise:

ssl_option no_sslv2
ssl_option no_sslv3
#ssl_option no_tlsv1
ssl_option cipher_server_preference

This configuration won’t work with IE6 on XP; hopefully nobody uses that anymore. The extremely long string at the end is the ciphersuite suggested by Mozilla for intermediate compatibility at the time of writing. For the full story, see here: