Tag Archives: example

High Performance Load Balancing Using DSR on a Raspberry Pi 2

TL;DR: Direct Server Return allows a Raspberry Pi or other low-end equipment to balance load at performance impossible to achieve using other methods.

Test conditions

Pi directly connected to laptop with Ethernet.

Laptop running Debian VM in Virtual Box with adapter 1 on NAT and adapter 2 on Ethernet (bridged)

Two more VM:s in VirtualBox on the same laptop to be used as test targets, connected in the same way as the first.

One tricky aspect of this test is that DSR requires a dedicated network interface, and the Pi only has one. This means that everything needs to be set up with the interface configured normally, and then the interface must be reconfigured and the test controlled from the console.

The Debian VM is temporarily set up as gateway to the outside world.

root@debian:~# ifconfig eth1 192.168.100.1/24
root@debian:~# echo 1 > /proc/sys/net/ipv4/ip_forward
root@debian:~# iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

And the Pi is temporarily set up with this configuration:

Pi eth0 = 192.168.100.2/24
GW = 192.168.100.1

ping 8.8.8.8 from pi

Works!

Install Pen

Make sure Raspbian on the Pi is up to date.

apt-get update
apt-get upgrade

Continue with instructions from the Wiki.

apt-get install automake autoconf gcc git
mkdir Git
cd Git
git clone https://github.com/UlricE/pen.git
cd pen
aclocal
automake --add-missing
autoconf
./configure
make

Verify the installation

./pen -dfU 53 8.8.8.8:53

And from the Debian VM:

root@debian:~# dig @192.168.100.2 +short siag.nu
194.9.95.65

Pen is now confirmed to work.

Let’s try one more. There is an Apache server running on the Debian VM.

root@raspberrypi:~/Git/pen# ./pen -df 80 192.168.100.1

root@debian:~# lynx -dump http://192.168.100.2/
It works!

This is the default web page for this server.

The web server software is running but no content has been added, yet.

Configure the Pi for DSR

Everything seems good to go. We can now reconfigure eth0 on the Pi.

ifconfig eth0 0.0.0.0

./pen -df -O "dsr_if eth0" -r 192.168.100.10:0 192.168.100.3 192.168.100.4

This means that we intend to forward any TCP traffic with destination address 192.168.100.10 to the two servers 192.168.100.3 and 192.168.100.4, load balanced using round robin.

Those two addresses exist on two additional Debian VM:s on the same laptop. Like the first one they each have eth0 connected to NAT and eth1 connected to wired ethernet.

Set up test targets

Both VM:s need a loopback interface configured with the virtual address
which they must mot tell anyone about:

ifconfig lo:1 192.168.100.10/32
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore

Restart Apache to make sure it listens on the new address:

service apache2 restart

Finally verify that we can access Apache on all addresses:

root@debian:~# lynx -dump http://192.168.100.3/
It works!

This is the default web page for this server.

The web server software is running but no content has been added, yet.
root@debian:~# lynx -dump http://192.168.100.4/
It works!

This is the default web page for this server.

The web server software is running but no content has been added, yet.
root@debian:~# lynx -dump http://192.168.100.10/
It works!

This is the default web page for this server.

The web server software is running but no content has been added, yet.

And here is one of the main reasons for wanting to use DSR:

root@debian:~# ab -n 1000 -c 20 http://192.168.100.10/1000k
This is ApacheBench, Version 2.3 <$Revision: 1604373 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.100.10 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests

Server Software: Apache/2.2.22
Server Hostname: 192.168.100.10
Server Port: 80

Document Path: /1000k
Document Length: 1024000 bytes

Concurrency Level: 20
Time taken for tests: 13.427 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 1024235000 bytes
HTML transferred: 1024000000 bytes
Requests per second: 74.48 [#/sec] (mean)
Time per request: 268.532 [ms] (mean)
Time per request: 13.427 [ms] (mean, across all concurrent requests)
Transfer rate: 74496.13 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 1 7 8.3 6 239
Processing: 74 258 139.7 210 1081
Waiting: 0 11 22.4 8 239
Total: 77 265 140.0 217 1087

Percentage of the requests served within a certain time (ms)
50% 217
66% 239
75% 269
80% 390
90% 439
95% 517
98% 691
99% 782
100% 1087 (longest request)

The document 1000k is a dummy file containing 1024000 zeroes. Fetching it at 74.48 requests per second corresponds to a bandwidth of 610 Mbps, a speed physically impossible to achieve through the Pi’s Fast Ethernet interface, but easily achieved using DSR since the return traffic bypasses the load balancer completely. CPU usage on the Pi hovered at 15-20% during the test.

Facebooktwittergoogle_plusredditpinterestlinkedinmail

Load Balancing UDP with Pen 0.25.0 – Replies

The upcoming Pen 0.25.0 release will handle load balancing for UDP protocols where the client expects a reply from the server. An example of this is DNS.

Here we let Pen listen on UDP port 8000 and forward all requests to Google’s public DNS servers 8.8.8.8 and 8.8.4.4.


ulric@debian:~/Projekt/pen$ ./pen -dfrU -S 2 8000 8.8.8.8:53 8.8.4.4:53 2>&1 | more
2014-08-01 14:55:33: pen_aton(0.0.0.0, 0x7fff3b0face0)
2014-08-01 14:55:33: family = 2
2014-08-01 14:55:33: socktype = 1
2014-08-01 14:55:33: protocol = 6
2014-08-01 14:55:33: addrlen = 16
2014-08-01 14:55:33: sockaddr = 0x2141250
2014-08-01 14:55:33: canonname = (null)
2014-08-01 14:55:33: local address=[0.0.0.0:8000]
2014-08-01 14:55:33: n = 2, address = 8.8.8.8, pno = 53, maxc1 = 0, hard = 0, weight = 0, prio = 0, proto = udp
2014-08-01 14:55:33: pen_aton(8.8.8.8, 0x21478b8)
2014-08-01 14:55:33: family = 2
2014-08-01 14:55:33: socktype = 1
2014-08-01 14:55:33: protocol = 6
2014-08-01 14:55:33: addrlen = 16
2014-08-01 14:55:33: sockaddr = 0x21412d0
2014-08-01 14:55:33: canonname = (null)
2014-08-01 14:55:33: n = 2, address = 8.8.4.4, pno = 53, maxc1 = 0, hard = 0, weight = 0, prio = 0, proto = udp
2014-08-01 14:55:33: pen_aton(8.8.4.4, 0x2147968)
2014-08-01 14:55:33: family = 2
2014-08-01 14:55:33: socktype = 1
2014-08-01 14:55:33: protocol = 6
2014-08-01 14:55:33: addrlen = 16
2014-08-01 14:55:33: sockaddr = 0x21412d0
2014-08-01 14:55:33: canonname = (null)
2014-08-01 14:55:33: pen 0.25.0 starting
2014-08-01 14:55:33: servers:
2014-08-01 14:55:33: 0 8.8.8.8:53:0:0:0:0
2014-08-01 14:55:33: 1 8.8.4.4:53:0:0:0:0
2014-08-01 14:55:33: read_cfg((null))
2014-08-01 14:55:33: mainloop_select()
2014-08-01 14:55:33: After zero
2014-08-01 14:55:33: After setting fd_sets
2014-08-01 14:55:33: w_read 3 is set

Pen is now waiting for a client.

On another host, we make a DNS query for a resource on the Internet:


ulric@qvp2:~$ dig @192.168.0.174 -p 8000 siag.nu

; <<>> DiG 9.9.3-rpz2+rl.13214.22-P2-Ubuntu-1:9.9.3.dfsg.P2-4ubuntu3 <<>> @192.168.0.174 -p 8000 siag.nu
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 47910 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;siag.nu. IN A ;; ANSWER SECTION: siag.nu. 299 IN A 194.9.95.65 ;; Query time: 74 msec ;; SERVER: 192.168.0.174#8000(192.168.0.174) ;; WHEN: Fri Aug 01 14:56:13 CEST 2014 ;; MSG SIZE rcvd: 52

The reply looks fine. What do we have in Pen's debug output?


2014-08-01 14:56:13: After select
2014-08-01 14:56:13: w_read 3 is set
2014-08-01 14:56:13: add_client: received 36 bytes from client
2014-08-01 14:56:13: match_acl_ipv4(0, 33685514)
2014-08-01 14:56:13: Trying server 1 at time 1406897773
2014-08-01 14:56:13: match_acl_ipv4(0, 33685514)
2014-08-01 14:56:13: Connecting to 8.8.4.4
2014-08-01 14:56:13: Family: AF_INET
2014-08-01 14:56:13: Port: 53
2014-08-01 14:56:13: Address: 8.8.4.4
2014-08-01 14:56:13: Successful connect to server 1
2014-08-01 14:56:13: Client 10.0.2.2 has index 0 and server 1
2014-08-01 14:56:13: store_conn: conn = 0, upfd = 4, downfd = 3, connections_used = 1
2014-08-01 14:56:13: add_client: wrote 36 bytes to socket 4
2014-08-01 14:56:13: checking sockets from open connections
2014-08-01 14:56:13: checking connection slot 0
2014-08-01 14:56:13: from 10.0.2.2
2014-08-01 14:56:13: to 8.8.4.4
2014-08-01 14:56:13: After zero
2014-08-01 14:56:13: interested in reading data from upstream socket 4 of connection 0
2014-08-01 14:56:13: After setting fd_sets
2014-08-01 14:56:13: w_read 3 is set
2014-08-01 14:56:13: w_read 4 is set
2014-08-01 14:56:13: After select
2014-08-01 14:56:13: w_read 4 is set
2014-08-01 14:56:13: checking sockets from open connections
2014-08-01 14:56:13: checking connection slot 0
2014-08-01 14:56:13: from 10.0.2.2
2014-08-01 14:56:13: to 8.8.4.4
2014-08-01 14:56:13: want to read from upstream socket 4 of connection 0
2014-08-01 14:56:13: copy_down sending 52 bytes to socket 3
2014-08-01 14:56:13: close_conn: Closing connection 0 to server 1; connections_used = 0
2014-08-01 14:56:13: Read 0 from client, wrote 0 to server
2014-08-01 14:56:13: Read 0 from server, wrote 0 to client
2014-08-01 14:56:13: After zero
2014-08-01 14:56:13: After setting fd_sets
2014-08-01 14:56:13: w_read 3 is set

Let's try it again:


ulric@qvp2:~$ dig @192.168.0.174 -p 8000 siag.nu

; <<>> DiG 9.9.3-rpz2+rl.13214.22-P2-Ubuntu-1:9.9.3.dfsg.P2-4ubuntu3 <<>> @192.168.0.174 -p 8000 siag.nu
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 863 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;siag.nu. IN A ;; ANSWER SECTION: siag.nu. 210 IN A 194.9.95.65 ;; Query time: 53 msec ;; SERVER: 192.168.0.174#8000(192.168.0.174) ;; WHEN: Fri Aug 01 14:57:42 CEST 2014 ;; MSG SIZE rcvd: 52

Same result. What does Pen have to say about it?


2014-08-01 14:57:42: After select
2014-08-01 14:57:42: w_read 3 is set
2014-08-01 14:57:42: add_client: received 36 bytes from client
2014-08-01 14:57:42: match_acl_ipv4(0, 33685514)
2014-08-01 14:57:42: Trying server 0 at time 1406897862
2014-08-01 14:57:42: match_acl_ipv4(0, 33685514)
2014-08-01 14:57:42: Connecting to 8.8.8.8
2014-08-01 14:57:42: Family: AF_INET
2014-08-01 14:57:42: Port: 53
2014-08-01 14:57:42: Address: 8.8.8.8
2014-08-01 14:57:42: Successful connect to server 0
2014-08-01 14:57:42: Client 10.0.2.2 has index 0 and server 0
2014-08-01 14:57:42: store_conn: conn = 0, upfd = 4, downfd = 3, connections_used = 1
2014-08-01 14:57:42: add_client: wrote 36 bytes to socket 4
2014-08-01 14:57:42: checking sockets from open connections
2014-08-01 14:57:42: checking connection slot 0
2014-08-01 14:57:42: from 10.0.2.2
2014-08-01 14:57:42: to 8.8.8.8
2014-08-01 14:57:42: After zero
2014-08-01 14:57:42: interested in reading data from upstream socket 4 of connection 0
2014-08-01 14:57:42: After setting fd_sets
2014-08-01 14:57:42: w_read 3 is set
2014-08-01 14:57:42: w_read 4 is set
2014-08-01 14:57:42: After select
2014-08-01 14:57:42: w_read 4 is set
2014-08-01 14:57:42: checking sockets from open connections
2014-08-01 14:57:42: checking connection slot 0
2014-08-01 14:57:42: from 10.0.2.2
2014-08-01 14:57:42: to 8.8.8.8
2014-08-01 14:57:42: want to read from upstream socket 4 of connection 0
2014-08-01 14:57:42: copy_down sending 52 bytes to socket 3
2014-08-01 14:57:42: close_conn: Closing connection 0 to server 0; connections_used = 0
2014-08-01 14:57:42: Read 0 from client, wrote 0 to server
2014-08-01 14:57:42: Read 0 from server, wrote 0 to client
2014-08-01 14:57:42: After zero
2014-08-01 14:57:42: After setting fd_sets
2014-08-01 14:57:42: w_read 3 is set

This time we got the reply from the other server. So round-robin load balancing works. If we hadn't used the -r option, Pen would have stayed with 8.8.4.4, because that was what worked the last time.

Facebooktwittergoogle_plusredditpinterestlinkedinmail

Using Pen as an Address Family Adapter

This is for the soon to be released 0.23 version of Pen.

Let’s say we have a service listening on a Unix domain socket and want to make it available over a network. The Swiss Army Pen Knife can do that, of course.

First we create such a service:

$ pen -dfS 1 ./ssh localhost:22

2014-04-11 17:41:56: n = 2, address = 127.0.0.1, pno = 22, maxc1 = 0, hard = 0, weight = 0, prio = 0, proto = tcp
2014-04-11 17:41:56: pen_aton(127.0.0.1, 0xac3028)
2014-04-11 17:41:56: family = 2
2014-04-11 17:41:56: socktype = 1
2014-04-11 17:41:56: protocol = 6
2014-04-11 17:41:56: addrlen = 16
2014-04-11 17:41:56: sockaddr = 0xac45e0
2014-04-11 17:41:56: canonname = (null)
2014-04-11 17:41:56: servers:
2014-04-11 17:41:56:  0 127.0.0.1:22:0:0:0:0
2014-04-11 17:41:56: mainloop_select()

Here, we wait for connections to ./ssh and forward them to our real ssh daemon. The “-dfS 1” part is to turn on debugging, stay in the foreground so we can see what’s going on and only use a single backend server (for less output clutter).

Now we have ssh listening on the ./ssh socket. This part is only necessary in order to create a test target for this demonstration.

Then we create the network-to-unix-socket part:

$ pen -dfS 1 :::2222 ./ssh

2014-04-11 17:42:23: pen_aton(::, 0x7fff9e734780)
2014-04-11 17:42:23: family = 10
2014-04-11 17:42:23: socktype = 1
2014-04-11 17:42:23: protocol = 6
2014-04-11 17:42:23: addrlen = 28
2014-04-11 17:42:23: sockaddr = 0x1c143b0
2014-04-11 17:42:23: canonname = (null)
2014-04-11 17:42:23: local address=[:::2222]
2014-04-11 17:42:23: n = 1, address = ./ssh, pno = 0, maxc1 = 0, hard = 0, weight = 0, prio = 0, proto = tcp
2014-04-11 17:42:23: pen_aton(./ssh, 0x1c1aa18)
2014-04-11 17:42:23: servers:
2014-04-11 17:42:23:  0 ./ssh:1:0:0:0:0
2014-04-11 17:42:23: mainloop_select()

Here, we wait for connections to port 2222 on the address ::. On Linux, that will make our single pen instance accept connections to any ipv6 or ipv4 address.

Finally:

$ ssh -p 2222 localhost
ulric@localhost's password:
Linux debian 3.12-1-amd64 #1 SMP Debian 3.12.9-1 (2014-02-01) x86_64
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
You have new mail.
Last login: Fri Apr 11 17:30:04 2014 from localhost

So that obviously works. In the first debugging windows, we have:

 

2014-04-11 17:42:47: Unix acl:s not implemented
2014-04-11 17:42:47: Client  has index -1
2014-04-11 17:42:47: lookup_client returns -1
2014-04-11 17:42:47: Trying server 0 at time 1397230967
2014-04-11 17:42:47: Unix acl:s not implemented
2014-04-11 17:42:47: Connecting to 127.0.0.1
2014-04-11 17:42:47: Family: AF_INET
2014-04-11 17:42:47: Port: 22
2014-04-11 17:42:47: Address: 127.0.0.1
2014-04-11 17:42:47: Successful connect to server 0
2014-04-11 17:42:47: Client  has index 0 and server 0
2014-04-11 17:42:47: store_conn: connections_used = 1

“Unix acl:s not implemented” means that unlike ipv4 and ipv6 connections, Pen has no built in restrictions for local sockets. Since they follow regular file permissions, you can use regular Unix tools (chmod et al) to restrict access.

And in the second:

2014-04-11 17:42:47: match_acl_ipv6(0, 2658351864)
2014-04-11 17:42:47: Client ::1 has index -1
2014-04-11 17:42:47: lookup_client returns -1
2014-04-11 17:42:47: Trying server 0 at time 1397230967
2014-04-11 17:42:47: match_acl_ipv4(0, 2658351864)
2014-04-11 17:42:47: Connecting to ./ssh
2014-04-11 17:42:47: Family: AF_UNIX
2014-04-11 17:42:47: Path: ./ssh
2014-04-11 17:42:47: Successful connect to server 0
2014-04-11 17:42:47: Client ::1 has index 0 and server 0
2014-04-11 17:42:47: store_conn: connections_used = 1
Facebooktwittergoogle_plusredditpinterestlinkedinmail

Unix Domain Sockets

One of the new features in 0.22 is Unix domain sockets. Originally this came up as an idea to provide more fine-grained restrictions on who can use the control socket, but the control socket and the “main” socket are created by mostly the same code, so now it is possible to make Apache answer on a Unix domain socket, something which Apache cannot do by itself:

pen ./websocket localhost:80

Or for ssh:

pen ./sshd localhost:22

Now you just need a web browser and an ssh client which can connect to the local socket.

Facebooktwittergoogle_plusredditpinterestlinkedinmail