Interesting observation made while testing direct server return through Pen in VirtualBox.
Using e1000 drivers will send 1514 byte frames. Pen receives the frames, scribbles a bit on the headers and puts them back on the network. Iperf performance through Pen is pretty decent, 200 Mbps. Direct iperf without Pen in between is 400 Mbps. That makes sense, total throughput is limited by the host running all three virtual servers.
With virtio drivers, iperf performance without Pen is 1 Gbps. But performance with Pen drops to < 100 kbps! What's up with that? It turns out that the virtio nic drivers support TCP Segmentation Offload and will send oversized frames, expecting the physical nic to do the frame segmentation. With an entirely virtual network, no segmentation takes place. As a result, Pen receives oversized frames which it can't forward. The frames are dropped and TCP gets grumpy.
To get rid of this behaviour, TSO must be turned off on the iperf client:
ethtool -K eth1 tso off
However, this makes direct iperf performance between two vm:s drop from 1 Gbps to 365 Mbps. Also, iperf performance through Pen drops from 200 Mbps to 60 Mbps. So virtio nics are actually slower for this purpose than software emulated e1000.