Tag Archives: security

Pen 0.32.0 released

Available here:


And also here:


Pen 0.32.0 adds tarpit functionality to the Direct Server Return mode. The purpose of tarpitting is to make network scanning harder by producing lots of false positives.

Full list of changes from 0.31.1:

151123 Released 0.32.0.

151120 Added tarpit test case to testsuite.sh.

151117 Tarpit functionality to be used with the DSR mode.

151112 pen.1: removed obsolete -S option, updated defaults for -x and -L.


Tarpit support in Pen

Pen 0.32 will have built-in tarpit support in its Direct Server Return mode. The feature is enabled by specifying an access control list against which incoming requests are matched. Matching destination addresses will make Pen do two things:

1. Reply to ARP requests to such addresses.
2. Reply to TCP SYN with SYN+ACK.

The idea behind tarpitting is to slow down network scanning by giving lots of false positives. Pen does this with very little load and without having to manage any state.

Here is an example command line.

pen -df -O “acl 1 permit” -O “tarpit_acl 1” -O “dsr_if eth1”

Let’s go through that option by option.

acl 1 permit creates an entry in access list 1 which matches IP address All other IP addresses will be rejected.

tarpit_acl 1 makes Pen use access list 1, the one with as its sole entry, to match destination addresses.

dsr_if eth1 makes Pen use eth1 as the network interface where all direct server return processing is performed. is the address and port where Pen listens for legitimate requests. They will be forwarded to the backend servers. and are the backend servers. They have web servers listening on port 80 and IP address configured on a loopback interface. See the Wiki.

Let’s try making a legitimate request.

ulric@debtest:~/Git/pen$ curl

That worked fine. Frames from us go to Pen, Pen forwards them to one of the web servers, the web server replies directly to us. In Wireshark we see:


But what happens when we try the same thing on a tarpitted address?

ulric@debtest:~/Git/pen$ curl

It just hangs. We send SYN, Pen replies with SYN+ACK, we send ACK and think that the TCP handshake is done. So we send the HTTP request, which Pen ignores. We send it again. Pen ignores it again, and so on. Here’s what that looks like in Wireshark:


Access control lists are a very flexible way to control the tarpit functionality in Pen and have it tarpit every address in a subnet (except those that shouldn’t). As an example, think of a network with the following hosts: gateway web server 1 web server 2 load balanced address

The corresponding ACL would be created like this:

acl 1 deny
acl 1 deny
acl 1 deny
acl 1 deny

Anything Pen sees that is not destined for one of these addresses will be tarpitted.


How to get A+ on Qualys SSL Labs Test

This requires the version of Pen currently in Git, or 0.27.4 when that is released in a few days.


For this exercise, we’ll throw compatibility with older operating systems and browsers out and only focus on maxing out security.


First, we need a 4096 bit private key. In the following, replace “your.domain” with the real domain name you’re going to protect.

openssl genrsa -out your.domain.key 4096
openssl req -sha256 -new -key your.domain.key -out your.domain.csr

Your private key is in the file your.domain.key. The file your.domain.csr contains your certificate signing request, which needs to be sent to your certification authority. The details of that procedure is different depending on the CA, but should result in you having your new certificate in your possession. Save the certificate as your.domain.crt.

The final piece of information you need is the CA’s certificate, which the CA will provide. Save the certificate as intermediate.crt.

Assuming you managed to cobble together all these files in the directory /etc/pen, the certificate installation is now finished.

Protocol Support

This is easy. Nobody supports SSL 2.0 anymore. SSL 3.0 is only for IE6 on Windows XP, a dwindling user base. TLS 1.0 is still acceptable, but this is not an exercise in acceptability (or compatibility). Throw out everything but TLS 1.2 by putting the following in /etc/pen/https.cfg:

ssl_option no_sslv2
ssl_option no_sslv3
ssl_option no_tlsv1
ssl_option no_tlsv1.1

Cipher Strength

We want ECDHE support for perfect forward secrecy, we want 256 bits encryption, and we want to prefer the best ciphers. These lines in /etc/pen/https.cfg provide that:

ssl_option cipher_server_preference

Strict Transport Security

The final piece of the puzzle is HSTS, which we accomplish by putting this in our Apache config:

Header always set Strict-Transport-Security "max-age=63072000; includeSubdomains; preload"

Finally, enable mod_headers and restart Apache:

a2enmod headers
service apache2 restart

Start Pen

The command line for Pen looks like this:

/usr/local/bin/pen -u pen -C /var/run/pen/https.ctl -F /etc/pen/https.cfg -p /var/run/pen/https.pid -K /etc/pen/your.domain.key -E /etc/pen/your.domain.crt -G /etc/pen/intermediate.crt -S 2 443

That’s quite a bit to type. If you’re using Systemd, like the CentOS system that was used for this example, here’s the full unit file to be installed into /usr/lib/systemd/system:

Description=Pen load balancer (https)
ExecStart=/usr/local/bin/pen -u pen -C /var/run/pen/https.ctl -F /etc/pen/https.cfg -p /var/run/pen/https.pid -K /etc/pen/your.domain.key -E /etc/pen/your.domain.crt -G /etc/pen/intermediate.crt -S 2 443

See this post for more on Pen and Systemd.


Pen 0.27.3 released

Available here:


And also here:


Several new configuration options dealing with securing SSL.

ssl_option no_sslv2 turns off SSL2. This has been the default for ages. Nobody should use SSL2 anymore.

ssl_option no_sslv3 turns off SSL3, sacrificing compatibility with Windows XP but also “sacrificing” the associated vulnerabilities.

ssl_option no_tlsv1 turns off TLS1, again sacrificing a bit of compatibility for a bit of security.

ssl_option cipher_server_preference Prefer the ciphers listed at the beginning of the cipher list (see next item).

ssl_ciphers CIPHERS Specify a list of ciphers to support. By default, Pen will use whatever OpenSSL thinks the default should be, and that list will be different depending on the version of OpenSSL and the options used when compiling OpenSSL.

See here for a suggested configuration with intermediate compatibility but still good security:

Perfect Forward Secrecy

The default maximum number of connections has been 256 since Pen’s inception in 2000. Today that is ridiculously conservative since Pen will gladly handle tens of thousands of connections on a Raspberry Pi:

The Great Load Balancer Shootout…
Let’s double that one more time

The default is now bumped to 500; still very conservative.

Full list of changes since 0.27.2:

150330 Added autoconf check that the ECDHE is available and not disabled.
Bumped default max connections and listen queue to 500.

150326 Support for ECDHE cipher suites.

150325 New commands ssl_option and ssl_ciphers to individually disable
insecure protocols and ciphers.

150324 Updated penctl.1 with the new command.

150322 New knob to tweak max number of pending nonblocking connection
attempts: pending_max N (default 100).


Perfect Forward Secrecy

One of the new features in 0.27.3 will be perfect forward secrecy for SSL.


Enabling perfect forward secrecy involves picking an up to date version of OpenSSL compiled with the right options and using the appropriate ciphersuite. Do note that older clients (Windows XP) are at odds with secure SSL configuration – there’s no way to get both right at the same time. This suggested configuration is a compromise:

ssl_option no_sslv2
ssl_option no_sslv3
#ssl_option no_tlsv1
ssl_option cipher_server_preference

This configuration won’t work with IE6 on XP; hopefully nobody uses that anymore. The extremely long string at the end is the ciphersuite suggested by Mozilla for intermediate compatibility at the time of writing. For the full story, see here:



Security and Pen

Prompted by this:


Summary: use non-default features but none of the security ones in Pen and you can end up with something not very secure. First I thought “why do that?” but realized that Debian ship Pen without a lot of configuration hints. So here are a few:

  • Don’t run Pen as root
  • Use a jail
  • Use access lists to limit access


Here’s what needs to be done to create a chroot jail for Pen and run it there as a non-root user. Start/stop script added.

useradd pen
mkdir -p /var/lib/pen/etc /var/lib/pen/tmp
chown pen /var/lib/pen/tmp
grep ^pen: /etc/passwd > /var/lib/pen/etc/passwd
cat << EOF > /var/lib/pen/etc/pen.cfg
acl 0 deny
control_acl 0
acl 1 deny
client_acl 1
server 0 address port 88
cat << EOF > /etc/init.d/pen

case "\$1" in
start )
        pen -j /var/lib/pen -C 10080 -F /etc/pen.cfg -p /tmp/pen.pid 8080
stop )
        kill \`cat /var/lib/pen/tmp/pen.pid\`
* )
        echo "Usage: \$0 start|stop"
chmod +x /etc/init.d/pen