Pen

Pen is a highly scalable, highly available, robust load balancer for tcp and udp based protocols such as dns, http or smtp. It allows several servers to appear as one to the outside and automatically detects servers that are down and distributes clients among the available servers. This gives high availability and scalable performance.

Pen supports SSL offloading, Transparent Reverse Proxy (preserving client IP address), Direct Server Return and GeoIP filtering.

Pen is known to scale to hundreds of thousands of connections on low-end hardware.

Recent(-ish) changelog excerpt

150818	Released 0.30.0.

150803	Added UDP mode for Direct Server Return.

150803	Updated configure.ac for compatibility with CentOS 6.

150725	Added #ifdef around SSLv3 initialization code in ssl, as
	suggested by jca@openbsd.org.

150608	Released 0.29.0.

150528	Transparent reverse proxy support for Linux, FreeBSD and OpenBSD.

150527	Allow the client table size to be updated on the fly. Default size still 2048.
	Allow the connection table size to be updated in the fly. Default still 500.
	See penctl.1, options clients_max and conn_max.

150526	Introduced the macro NO_SERVER to be used instead of -1 to signify
	error conditions and such.
	Removed the fixed server table size along with the -S option.

150525	Fixed cosmetic bug in startup code which required port to be specified
	on backend servers even if it was the same as the listening port.

Change log

The load balancing algorithm keeps track of clients and will try to send them back to the server they visited the last time. The client table has a number of slots (default 2048, settable through command-line arguments). When the table is full, the least recently used one will be thrown out to make room for the new one.

This is superior to a simple round-robin algorithm, which sends a client that connects repeatedly to different servers. Doing so breaks applications that maintain state between connections in the server, including most modern web applications.

When pen detects that a server is unavailable, it scans for another starting with the server after the most recently used one. That way we get load balancing and “fair” failover for free.

Correctly configured, pen can ensure that a server farm is always available, even when individual servers are brought down for maintenance or reconfiguration. The final single point of failure, pen itself, can be eliminated by running pen on several servers, using vrrp to decide which is active.
Description of redundancy using vrrpd on Linux
The Ultimate Cheapskate Cluster

Penlogd

A side-effect of the load-balancing is that several logfiles are produced, and all accesses seem to come from the load balancer. The program penlogd solves this problem by merging pen’s log file with the ones produced by the web servers. See penlogd(1) and penlog(1) for details.

Webstats

Pen emits statistics when it receives a USR1 signal, but the output can be hard to interpret. The cgi script webstats can be used in conjunction with the -w option to pen to get statistics in HTML format instead.

New: Take a look here to see what the latest Pen status is here at siag.nu. This is not “live” data, but generated from this cron job every 10 minutes:

2,12,22,32,42,52 * * * *        kill -USR1 `cat /var/run/pen.pid`

Note that there are usually few active connections. This is because of the way the web works: the client connects, the server sends data, the client disconnects. The whole transaction is over in seconds.

Portability

This load balancer is known to work on FreeBSD, Linux, HP-UX and Solaris. Other Unixes should work as well, possibly requiring trivial changes. Success stories or problem reports are welcome.

It runs on Windows, too.
And on MacOS X.

 

Installation

Actually, the easiest way to install Pen nowadays is to get it from one of the distributions that package it. On e.g. Debian or Ubuntu the entire process is:

	apt-get install pen

If you still want to install from source because you need features not built into the packages, here’s how. Type:

	./configure
        make
        make install

By default the programs are installed in /usr/local/bin. This can be changed like this example:

        ./configure --prefix=/usr

to install into /usr/bin instead.

Download source

Contributed start scripts

Links

Github source repository

Hercules Load Balancer Virtual Appliance, a VMware virtual machine based on Pen (“repackaged” version here)

Penbw, Pen backend watcher, monitors and blacklists backends automatically in case of a failure

Pre-compiled packages for Solaris

A redundant load-balancing firewall system, using FreeBSD

Zen load balancer, a load balancer appliance based on Pen.

Freshmeat project page

Load balancing UDP on EC2

Fedora package

Debian package

 

20 thoughts on “Pen

  1. I’m trying to use pen to make a “pure” fail over server. So I use this
    pen -fdd -e www1:389 389 www2:389

    However, when i down the service in www2, it seems like the emergency server www1 is not take over the service. The “foreground” log as show:

    2014-05-08 16:16:47: match_acl(0, 16777343)
    2014-05-08 16:16:47: Client 127.0.0.1 has index 0
    2014-05-08 16:16:47: lookup_client returns 0
    2014-05-08 16:16:47: Trying server 0 at time 1399537007
    2014-05-08 16:16:47: match_acl(0, 16777343)
    2014-05-08 16:16:47: Server 0 failed, retry in 30 sec: Connection refused
    2014-05-08 16:16:47: Trying server 1 at time 1399537007
    2014-05-08 16:16:47: No port for you!
    2014-05-08 16:16:47: Trying server 2 at time 1399537007
    2014-05-08 16:16:47: No port for you!
    2014-05-08 16:16:47: Trying server 3 at time 1399537007
    2014-05-08 16:16:47: No port for you!
    2014-05-08 16:16:47: Trying server 4 at time 1399537007
    2014-05-08 16:16:47: No port for you!
    2014-05-08 16:16:47: Trying server 5 at time 1399537007
    2014-05-08 16:16:47: No port for you!
    2014-05-08 16:16:47: Trying server 6 at time 1399537007
    2014-05-08 16:16:47: No port for you!
    2014-05-08 16:16:47: Trying server 7 at time 1399537007
    2014-05-08 16:16:47: No port for you!
    2014-05-08 16:16:47: Trying server 8 at time 1399537007
    2014-05-08 16:16:47: No port for you!
    2014-05-08 16:16:47: Trying server 9 at time 1399537007
    2014-05-08 16:16:47: No port for you!
    2014-05-08 16:16:47: Trying server 10 at time 1399537007
    2014-05-08 16:16:47: No port for you!
    2014-05-08 16:16:47: Trying server 11 at time 1399537007
    2014-05-08 16:16:47: No port for you!
    2014-05-08 16:16:47: Trying server 12 at time 1399537007
    2014-05-08 16:16:47: No port for you!
    2014-05-08 16:16:47: Trying server 13 at time 1399537007
    2014-05-08 16:16:47: No port for you!
    2014-05-08 16:16:47: Trying server 14 at time 1399537007
    2014-05-08 16:16:47: No port for you!
    2014-05-08 16:16:47: Trying server 15 at time 1399537007
    2014-05-08 16:16:47: No port for you!
    2014-05-08 16:16:47: Trying server 0 at time 1399537007
    2014-05-08 16:16:47: Server 0 is blacklisted

    is this as expected?

    thank you

  2. The emergency server isn’t entirely intuitive. You actually need to add the -s flag too, like this:

    /pen -fds -S 1 -e 127.0.0.1:22 2222 127.0.0.1:23

    And:

    ulric@debian:~/Projekt/pen/Zippar/pen-0.21.1$ telnet localhost 2222
    Trying ::1…
    Trying 127.0.0.1…
    Connected to localhost.
    Escape character is ‘^]’.
    SSH-2.0-OpenSSH_6.4p1 Debian-2

    Debugging output:

    2014-04-24 12:32:21: local address=[0.0.0.0:2222]
    2014-04-24 12:32:21: n = 2, address = 127.0.0.1, pno = 23, maxc1 = 0, hard = 0, weight = 0, prio = 0, proto = tcp
    2014-04-24 12:32:21: n = 2, address = 127.0.0.1, pno = 22, maxc1 = 0, hard = 0, weight = 0, prio = 0, proto = tcp
    2014-04-24 12:32:21: servers:
    2014-04-24 12:32:21: 0 127.0.0.1:23:0:0:0:0
    2014-04-24 12:32:21: mainloop_select()
    2014-04-24 12:32:23: match_acl_ipv4(0, 16777343)
    2014-04-24 12:32:23: Client 127.0.0.1 has index -1
    2014-04-24 12:32:23: lookup_client returns -1
    2014-04-24 12:32:23: Using emergency server
    2014-04-24 12:32:23: Trying server 1 at time 1398335543
    2014-04-24 12:32:23: match_acl_ipv4(0, 16777343)
    2014-04-24 12:32:23: Successful connect to server 1
    2014-04-24 12:32:23: Client 127.0.0.1 has index 0 and server -1
    2014-04-24 12:32:23: store_conn: connections_used = 1
    2014-04-24 12:32:27: close_conn: connections_used = 0

  3. How to or is it possible to pass source IP with UDP requests to servers behind pen LB? I am only able to get the ip of pen LB server, not source IP. Tried -H option, didn’t work.

  4. Hello. I’m trying to use pen as an UDP Loadbalancer for two FreeRadius Servers.
    But if I block access to one of the two servers pen does not switch to the second server.It remains stuck, even if I re-enable the access to it.
    Here ist my pen commandline:

    pen -S 2 -d -f -W -r -U 10.10.100.42:1812 10.10.100.10:1812:100:0:10:0 10.10.100.11:1812:100:0:0:0

    I block access to the first server with a simple ‘iptables -A OUTPUT -d 10.10.100.10 -j DROP’ on the same host where pen runs.

    It does not switch to the second. The debug output shows the following:
    2014-05-16 11:46:02: Trying server 0 at time 1400233562
    2014-05-16 11:46:02: match_acl_ipv4(0, 2500004106)
    2014-05-16 11:46:02: Successful connect to server 0
    2014-05-16 11:46:02: Client 10.9.3.149 has index 1 and server 0
    2014-05-16 11:46:02: store_conn: connections_used = 1
    2014-05-16 11:46:02: close_conn: connections_used = 0
    2014-05-16 11:46:07: alarm_handler(14)

    Am I doing something stupid on the command line?
    Thanks for any hint,
    Thomas Plant

    1. Nothing stupid, just that udp works differently from tcp. For tcp, pen detects that the connect fails and blacklists the server. For udp, connect just stores the address, so pen doesn’t know if the server is available or not.

      You can use a separate script to monitor the servers and let that script blacklist unavailable servers using penctl.

  5. Holy…. yes UDP….stateless…stupid me anyway. Fortunately it’s friday 🙂

    I’ll write a test script to check if server is available and blacklist as apropriate.

    Thanks for the reply.

  6. UDP support works but seems to have trouble under load. Running it on c2large instance and it has problems with more than 1-2 messages per second… pen is using 100% of one of the cores (load 1) and keeps on dropping messages.

  7. To clarify my last comment, the lb receives about 100 udp messages per sec (GELF UDP), but the backend servers receive 1-2 udp messages per sec.

    1. Not sure why this would happen, the load balancing code for udp and tcp is much the same and pen shouldn’t have any trouble with that kind of load.

      I will try and see if I can replicate the problem.

      1. Also, the udp traffic that does get routed to the backend servers seems to be corrupted. Example from our load environment (note this is m1.small instance.) This is the traffic that flows from the Pen running on 10.230.1.49:12201 to one of the two backend servers (10.230.1.45:12201)

        [root@load-logsvc ~]# tcpdump -vvv -n dst host 10.230.1.45
        tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
        15:33:40.011872 IP (tos 0x0, ttl 64, id 53334, offset 0, flags [DF], proto UDP (17), length 501)
        10.230.1.49.54415 > 10.230.1.45.12201: [bad udp cksum c71b!] UDP, length 473
        15:33:45.019571 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.230.1.45 tell 10.230.1.49, length 28
        15:34:15.000248 IP (tos 0x0, ttl 64, id 22786, offset 0, flags [DF], proto UDP (17), length 928)
        10.230.1.49.56206 > 10.230.1.45.12201: [bad udp cksum 299e!] UDP, length 900
        15:34:50.002817 IP (tos 0x0, ttl 64, id 57789, offset 0, flags [DF], proto UDP (17), length 427)
        10.230.1.49.49422 > 10.230.1.45.12201: [bad udp cksum 5524!] UDP, length 399
        15:34:55.009557 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.230.1.45 tell 10.230.1.49, length 28

        Sample of the incoming traffic:

        15:26:18.737997 IP 10.18.74.197.50715 > 10.230.1.49.12201: UDP, length 521
        15:26:18.737999 IP 10.18.74.197.50715 > 10.230.1.49.12201: UDP, length 471
        15:26:18.738000 IP 10.18.74.197.50715 > 10.230.1.49.12201: UDP, length 522
        15:26:18.738002 IP 10.18.74.197.50715 > 10.230.1.49.12201: UDP, length 475
        15:26:18.738003 IP 10.18.74.197.50715 > 10.230.1.49.12201: UDP, length 527
        15:26:18.738005 IP 10.18.74.197.50715 > 10.230.1.49.12201: UDP, length 522
        15:26:18.738007 IP 10.18.74.197.50715 > 10.230.1.49.12201: UDP, length 399
        15:26:18.738008 IP 10.18.74.197.50715 > 10.230.1.49.12201: UDP, length 522
        15:26:18.738010 IP 10.18.74.197.50715 > 10.230.1.49.12201: UDP, length 526
        15:26:18.738017 IP 10.18.74.197.50715 > 10.230.1.49.12201: UDP, length 525
        15:26:18.738019 IP 10.18.74.197.50715 > 10.230.1.49.12201: UDP, length 523
        15:26:18.738023 IP 10.18.74.197.50715 > 10.230.1.49.12201: UDP, length 470
        15:26:18.738458 IP 10.18.74.197.50715 > 10.230.1.49.12201: UDP, length 524
        15:26:18.738967 IP 10.18.74.197.50715 > 10.230.1.49.12201: UDP, length 522
        15:26:19.252620 IP 10.230.1.61.33008 > 10.230.1.49.12201: UDP, length 199
        15:26:19.258547 IP 10.230.1.61.52947 > 10.230.1.49.12201: UDP, length 209
        15:26:19.276033 IP 10.230.1.61.58391 > 10.230.1.49.12201: UDP, length 199
        15:26:19.285038 IP 10.230.1.61.58315 > 10.230.1.49.12201: UDP, length 229

        top – 15:36:46 up 5 days, 3:13, 1 user, load average: 1.47, 1.57, 1.54
        Tasks: 63 total, 2 running, 61 sleeping, 0 stopped, 0 zombie
        Cpu(s): 9.1%us, 43.2%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 47.6%st
        Mem: 3849284k total, 487536k used, 3361748k free, 117208k buffers
        Swap: 0k total, 0k used, 0k free, 150368k cached

        PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
        1009 root 20 0 44872 2784 940 R 56.7 0.1 4189:35 pen
        783 root 20 0 133m 39m 3192 S 42.2 1.1 3066:24 mcollectived

  8. First started using PEN about 4 years ago (Been working great BTW)… as I needed a solution for sticky session load balancing for internal users across 3 webservers. I am finding now that Dev is also using the PEN instance for customer web traffic trying to get to the same web servers… so I need to break these connections out, as they always come from one address pounding the hell out of one webserver daily due to the sticky sessions.

    I have setup another instance of PEN, specific for customer traffic, and need to have it do a round robin approach, to load across the 3 webservers from all connections coming from a single address. Is there a way to tell PEN to round robin? I know the reason I started using PEN was specifically for application need and internal employee access to web application requiring sticky sessions… however, I figured it could round robin as well.

    1. Indeed it can. By default, Pen uses sticky sessions because it is the safe thing to do, but you can easily turn that off with the -r option:

      pen -r [other options] 80 server1 server2 server3

    1. Yes. The reason for the high load and low throughput is that the main loops keeps trying to process non-existing data waiting on a socket. This will be addressed in 0.24.0 in a day or so.

  9. Hi Ulric,

    I’ve a few questions regarding penctl.
    I was trying to modify the list of backends onfly without restarting pen process.
    I managed to add a new backend by using penctl server option with a server number that doesn’t exist yet, which seems theoretically working fine.
    But I don’t know how to remove one of the backends on the fly using penctl command.
    is it possible ?

    One more issue I’ve tried to use penctl write command to write the conf file for pen as there is no documentation how to write it, but it didn’t write anything ?!

  10. Hello Ulric,

    I´d like to add a html-file to a balanced endpoint or redirect to e. g.

    host/example.html.

    Is it possible with pen?

    Thanks in advance
    Tom

Leave a Reply

Your email address will not be published. Required fields are marked *