Direct Server Return requires raw access to network traffic. There is no portable way to do that, so Pen supports it so far only on Linux and FreeBSD.
On Linux, the probe into the network stack is created pretty much like any other socket:
fd = socket(AF_PACKET, SOCK_RAW, IPPROTO_RAW);
It works literally like a probe, the packets are duplicated out and received both by the kernel and by the program creating the socket, Pen in this case. Care must be taken to prevent the kernel from interferring, for example by responding to TCP traffic.
Reading and writing is as usual:
recvfrom(fd, buf, MAXBUF, 0, NULL, NULL);
sendto(fd, b, len, 0, NULL, 0);
FreeBSD has a totally different solution called Netmap:
d = nm_open(ifname, NULL, 0, 0);
Here, d is not a socket but a “netmap descriptor” and reading and writing is done to rings of buffers, matching what is available in the network card. The regular network stack is cut off from the traffic. The details are hidden behind a pcap-like api:
nm_inject(d, b, len);
Now, wouldn’t it be fun to compare these two? Of course it would!
Two VMs are prepared with Apache and the address 192.168.100.1 on a loopback interface.
One VM runs ApacheBench like this:
ab -c 20 -n 10000 http://192.168.100.1/1000k
One Linux and one FreeBSD VM are prepared with the latest Pen from Git. On Linux the command line looks like:
sudo ./pen -df -O poll -O "dsr_if eth1" -S 2 -r 192.168.100.1:0 192.168.100.3 192.168.100.4
And on FreeBSD:
sudo ./pen -df -O poll -O "dsr_if em1" -S 2 -r 192.168.100.1:0 192.168.100.3 192.168.100.4
I.e. exactly the same, only the interface name differs.
Linux results here, ~0.9 Gbps:
And FreeBSD results, ~1.4 Gbps: