Monthly Archives: February 2015

Torturing the Pi 2 some more

Curiosity made me do it. How many connections will the Pi 2 withstand with a more realistic split between idle/active connections? The two previous tests used a 50/50 split, but in applications with massive numbers of long-lived connections it is common for only a few at a time to be active.

20 of 20 succeded
Idle Connections:             80000
Active Connections:           2000
Total Concurrency Level:      82000
Average time taken for tests: 1.487 seconds
Complete requests:            2000
Failed requests:              0
Total transferred:            868000 bytes
HTML transferred:             354000 bytes
Requests per second:          1344.98 [#/sec] (mean)
Transfer rate:                570.04 [Kbytes/sec] received

Pen 0.27.1 released

Available here:

And also here:

A bunch of bugfixes:

150219 In open_listener: check that the requested port is in range.
Fixed bug in dlist_insert.
Released 0.27.1.

150215 Even load distribution when a server is unavailable.

150212 Let pen save the settings for tcp_nodelay and tcp_fastclose.
Make flush_up and flush_down return the correct value on error.


Benchmarking on the RPi2

With reference to this test done a few days ago:

That the new Raspberry Pi 2 is faster than the old model is completely expected, but the difference is just astonishing:

20 of 20 succeded
Idle Connections:             12500
Active Connections:           12500
Total Concurrency Level:      25000
Average time taken for tests: 8.029 seconds
Complete requests:            12500
Failed requests:              0
Total transferred:            5425000 bytes
HTML transferred:             2212500 bytes
Requests per second:          1556.85 [#/sec] (mean)
Transfer rate:                659.83 [Kbytes/sec] received

Oh, and it was playing a movie at the same time. Not a glitch.


Benchmarking Pen on the Raspberry Pi

This test was conducted with 20 clients (from here on referred to as minions),
also acting as web servers with lighttpd. A Raspberry
Pi B revision 1 was used as load balancer. The revision 1 part is important,
because it only has 256 MB of RAM. The later revision 2 har 512 MB and would be
capable of handling more connections.

The test used 625 idle connections on each minion, along with 625 active connections
for a total of 20*(625+625) = 25000 connections to Pen. Since each connection *to*
Pen also results in a connection to the back-end server, that’s a total of 50000
TCP connections on the poor Pi.

20 of 20 succeded
 Idle Connections:             12500
 Active Connections:           12500
 Total Concurrency Level:      25000
 Average time taken for tests: 30.608 seconds
 Complete requests:            12500
 Failed requests:              0
 Total transferred:            5425000 bytes
 HTML transferred:             2212500 bytes
 Requests per second:          408.38 [#/sec] (mean)
 Transfer rate:                173.08 [Kbytes/sec] received

Pen 0.27.0 released

Available here:

And also here:

The 0.27.0 release focuses on doing more by doing less. The number of syscalls
required per transaction has been reduced significantly. On Linux, 100 ApacheBench
requests using all default Pen settings took 5689 syscalls on 0.26.1, but only
2819 on 0.27.0 (2413 with tcp_fastclose).

Another way of doing less has been to move certain operations from O(n) to O(1),
where n is the number of connections. That doesn’t matter much with the default
256 simultaneous connections, but when n is 100000, it makes a difference.

Two new configuration commands have seen the light of day. These can be set
on the command line at startup or dynamically with penctl.

tcp_nodelay|no tcp_nodelay

This controls the TCP_NODELAY option on sockets, which turns the Nagle algorithm
on or off on supported systems. May or may not make a difference.
Default is no tcp_nodelay.

tcp_fastclose up|down|both|off

With Pen being a proxy, one Pen connection is actually two TCP connections,
one to the client and one to the server. Normally, Pen waits for both client
and server to signal end of file before shutting down the connection.
For some protocols this is not necessary and it is safe to shut down the
connection when either end does. Default is off.

Finally, it is no longer required to run the configure script when
cross-compiling for Windows.

All ChangeLog entries since 0.26.1:

150212 Added with reasonable settings for Windows.

150211 Better detection and blacklisting of unavailable servers.

150209 New penctl commands:
tcp_nodelay sets TCP_NODELAY on sockets. Turn off with no tcp_nodelay.
tcp_fastclose closes both upstream and downstream sockets if one of them
closes theirs. Will take the values up, down, both or off (default).

150208 Rather than making a table of pending connections every time through
the main loop, keep them in a doubly linked list which is only updated
as needed. O(n) -> O(1).

150207 A bug in udp mode: after successful “connect”, do not event_add downfd,
because it is equal to listenfd and epoll_ctl doesn’t like that.

150206 Module kqueue.c updated.
Module poll.c: set unused fd:s to -1, or Solaris will say ENOSYS.

150205 Enable diagnostic messages by default in
Changed event bookkeeping from stateless to stateful.
Made keepalive optional and added “keepalive / no keepalive” penctl command.


Pen 0.26.0 released

Available here:

And also here:

The 0.26.0 release features quite a few improvements over 0.25.1. Some of
the most important ones are probably:

– Connecting to backend servers is now done in parallel with other operations.
This gives better overall performance for servers with a bit of latency
(i.e. most application servers).

– The option to turn off nonblocking socket operations has been removed.

– The accept queue length is now configurable, whereas it previously required
a recompile. This makes tuning easier.

– The number of incoming connections that are accepted at a time is also

– The configure script now automatically tries to use features that everybody
should want: poll, kqueue, epoll, openssl and geoip will be built if possible,
unless they are explicitly excluded. Before, it was necessary to write
a long list of “–with-foo –with-bar –with-baz” options.

– Pen has always used select as the default event management system. Support
for poll and kqueue has been available for many years, and has now been
joined by epoll. The default event management system is now kqueue for BSD,
epoll for Linux, poll for anything else if it is available and select
where it is the only option (i.e. Windows).

– Improved compatibility with Microsoft Windows. Pen can now be installed as
a native Windows service without using Cygwin as a compatibility layer.

Full list of changes since 0.25.1:

150204 Released 0.26.0.

150203 More sensible autoconfiguration defaults: poll, kqueue, epoll, openssl and geoip
are built if found unless explicitly excluded.
New event management defaults: kqueue, epoll, poll, select in that order.
New penctl commands: kqueue, epoll, poll, select.
New command line option: -O cmd where cmd is any penctl command.
E.g. -O select to use select instead of the compiled-in default.

150127 New penctl option “listen [address:]port” to allow listening address
to be changed on the fly or via a configuration file.
New pen options -i and -u to install and uninstall Pen as a Windows service.
See pen manpage.
Reduced default timeout to 3 seconds.

150126 New autoconf option –enable-debugging to enable debugging code.
Lots of fixes for compatibility with Windows.
Released 0.26.0beta2.

150123 Fixed bug in mainloop which kept trying to write 0 bytes.
MinGW port. Use to compile.

150121 Event management code broken out into select.c, poll.c, kqueue.c and epoll.c.

150113 New command-line option -m to accept multiple incoming connections in a batch.
New command-line option -q to set incoming pending connection queue length.

150112 Close upfd when failing over.

150109 Released 0.26.0beta1.
Adjusted debug logging levels.

150108 Started on epoll support for Linux.

150107 Rewrote output_net and output_file to take a variable number of arguments.
Handle timed out connection attempts in mainloop_kqueue.

150105 Fixed mainloop_kqueue.

150103 A lot of code broken out from mainloop_select into separate functions.
Fixed mainloop_poll.

150102 Bugfixes related to the new backend connection logic.

141229 Cleaned up and simplified add_client() and associated circuitry.
Connections to back end servers are now nonblocking and parallel.

141217 Removed the -n option and all code explicitly using blocking sockets.
Removed the -D option and the “delayed forward” feature.

141213 Renamed server and client fields in the conn, client and server structures
to better reflect what they are.
Restructured the add_client, store_client, store_conn and try_server