Load balancing load balancers and playing with affinity

Starting with Pen 0.33.0, it is possible to run several Pen instances on the same server and listening on the same IP address and port. The kernel takes care of distributing incoming connections to the instances.

To test this, start Pen like this. The VM test1 has two CPUs:

ulric@test1:~/Git/pen$ taskset -c 0 ./pen -ddf 5001 192.168.2.2 > log0 2>&1
ulric@test1:~/Git/pen$ taskset -c 0 ./pen -ddf 5001 192.168.2.3 > log1 2>&1

From a test client, run iperf in client mode:

ulric@debtest:~/Git/pen$ iperf -c test1 -P 20
------------------------------------------------------------
Client connecting to test1, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 22] local 192.168.1.1 port 37596 connected with 192.168.1.2 port 5001
[ 13] local 192.168.1.1 port 37587 connected with 192.168.1.2 port 5001
[ 4] local 192.168.1.1 port 37578 connected with 192.168.1.2 port 5001
[ 10] local 192.168.1.1 port 37584 connected with 192.168.1.2 port 5001
[ 6] local 192.168.1.1 port 37580 connected with 192.168.1.2 port 5001
[ 9] local 192.168.1.1 port 37583 connected with 192.168.1.2 port 5001
[ 7] local 192.168.1.1 port 37581 connected with 192.168.1.2 port 5001
[ 3] local 192.168.1.1 port 37577 connected with 192.168.1.2 port 5001
[ 14] local 192.168.1.1 port 37588 connected with 192.168.1.2 port 5001
[ 12] local 192.168.1.1 port 37586 connected with 192.168.1.2 port 5001
[ 15] local 192.168.1.1 port 37589 connected with 192.168.1.2 port 5001
[ 8] local 192.168.1.1 port 37582 connected with 192.168.1.2 port 5001
[ 5] local 192.168.1.1 port 37579 connected with 192.168.1.2 port 5001
[ 11] local 192.168.1.1 port 37585 connected with 192.168.1.2 port 5001
[ 17] local 192.168.1.1 port 37591 connected with 192.168.1.2 port 5001
[ 16] local 192.168.1.1 port 37590 connected with 192.168.1.2 port 5001
[ 18] local 192.168.1.1 port 37592 connected with 192.168.1.2 port 5001
[ 19] local 192.168.1.1 port 37593 connected with 192.168.1.2 port 5001
[ 20] local 192.168.1.1 port 37594 connected with 192.168.1.2 port 5001
[ 21] local 192.168.1.1 port 37595 connected with 192.168.1.2 port 5001
[ ID] Interval Transfer Bandwidth
[ 12] 0.0-10.0 sec 39.8 MBytes 33.3 Mbits/sec
[ 6] 0.0-10.0 sec 52.8 MBytes 44.2 Mbits/sec
[ 17] 0.0-10.0 sec 55.2 MBytes 46.3 Mbits/sec
[ 21] 0.0-10.0 sec 53.2 MBytes 44.6 Mbits/sec
[ 7] 0.0-10.0 sec 65.2 MBytes 54.6 Mbits/sec
[ 14] 0.0-10.0 sec 63.0 MBytes 52.8 Mbits/sec
[ 5] 0.0-10.0 sec 53.2 MBytes 44.6 Mbits/sec
[ 22] 0.0-10.0 sec 56.2 MBytes 47.0 Mbits/sec
[ 15] 0.0-10.0 sec 50.9 MBytes 42.6 Mbits/sec
[ 16] 0.0-10.0 sec 42.5 MBytes 35.6 Mbits/sec
[ 4] 0.0-10.0 sec 60.9 MBytes 50.9 Mbits/sec
[ 10] 0.0-10.0 sec 51.2 MBytes 42.8 Mbits/sec
[ 3] 0.0-10.0 sec 65.0 MBytes 54.3 Mbits/sec
[ 8] 0.0-10.0 sec 51.4 MBytes 42.9 Mbits/sec
[ 18] 0.0-10.0 sec 37.0 MBytes 30.9 Mbits/sec
[ 9] 0.0-10.0 sec 57.6 MBytes 48.1 Mbits/sec
[ 11] 0.0-10.0 sec 57.5 MBytes 48.0 Mbits/sec
[ 19] 0.0-10.1 sec 39.1 MBytes 32.6 Mbits/sec
[ 13] 0.0-10.1 sec 54.4 MBytes 45.3 Mbits/sec
[ 20] 0.0-10.1 sec 51.6 MBytes 42.9 Mbits/sec
[SUM] 0.0-10.1 sec 1.03 GBytes 880 Mbits/sec

192.168.2.2 and .3 run iperf in server mode.

[ 4] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 47827
[ 18] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 47826
[ 9] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 47837
[ 8] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 47832
[ 7] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 47831
[ 6] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 47829
[ 5] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 47828
[ 10] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 47843
[ 11] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 47844
[ 5] 0.0-10.1 sec 53.2 MBytes 44.2 Mbits/sec
[ 10] 0.0-10.1 sec 51.6 MBytes 42.8 Mbits/sec
[ 8] 0.0-10.1 sec 57.5 MBytes 47.6 Mbits/sec
[ 7] 0.0-10.1 sec 57.6 MBytes 47.7 Mbits/sec
[ 4] 0.0-10.1 sec 60.9 MBytes 50.3 Mbits/sec
[ 18] 0.0-10.2 sec 65.0 MBytes 53.6 Mbits/sec
[ 11] 0.0-10.2 sec 56.2 MBytes 46.4 Mbits/sec
[ 6] 0.0-10.2 sec 65.2 MBytes 53.7 Mbits/sec
[ 9] 0.0-10.4 sec 63.0 MBytes 51.0 Mbits/sec
[SUM] 0.0-10.4 sec 530 MBytes 430 Mbits/sec

and

[ 6] local 192.168.2.3 port 5001 connected with 192.168.2.1 port 59344
[ 5] local 192.168.2.3 port 5001 connected with 192.168.2.1 port 59343
[ 4] local 192.168.2.3 port 5001 connected with 192.168.2.1 port 59342
[ 9] local 192.168.2.3 port 5001 connected with 192.168.2.1 port 59339
[ 12] local 192.168.2.3 port 5001 connected with 192.168.2.1 port 59350
[ 11] local 192.168.2.3 port 5001 connected with 192.168.2.1 port 59349
[ 10] local 192.168.2.3 port 5001 connected with 192.168.2.1 port 59348
[ 8] local 192.168.2.3 port 5001 connected with 192.168.2.1 port 59347
[ 7] local 192.168.2.3 port 5001 connected with 192.168.2.1 port 59345
[ 14] local 192.168.2.3 port 5001 connected with 192.168.2.1 port 59354
[ 13] local 192.168.2.3 port 5001 connected with 192.168.2.1 port 59351
[ 12] 0.0-10.1 sec 37.0 MBytes 30.8 Mbits/sec
[ 10] 0.0-10.1 sec 42.5 MBytes 35.3 Mbits/sec
[ 13] 0.0-10.1 sec 39.1 MBytes 32.5 Mbits/sec
[ 5] 0.0-10.1 sec 51.2 MBytes 42.5 Mbits/sec
[ 8] 0.0-10.1 sec 50.9 MBytes 42.1 Mbits/sec
[ 4] 0.0-10.1 sec 51.4 MBytes 42.5 Mbits/sec
[ 9] 0.0-10.1 sec 52.8 MBytes 43.6 Mbits/sec
[ 7] 0.0-10.2 sec 54.4 MBytes 44.9 Mbits/sec
[ 6] 0.0-10.2 sec 39.8 MBytes 32.8 Mbits/sec
[ 11] 0.0-10.2 sec 55.2 MBytes 45.5 Mbits/sec
[ 14] 0.0-10.2 sec 53.2 MBytes 43.9 Mbits/sec
[SUM] 0.0-10.2 sec 528 MBytes 435 Mbits/sec

9 on one and 11 on the other, not perfect balance but pretty good. We can see the from the logs as well:

ulric@test1:~/Git/pen$ wc -l log[01]
95962 log0
96737 log1
192699 total

Now try this with the Pen instances on separate CPUs.

ulric@test1:~/Git/pen$ taskset -c 0 ./pen -ddf 5001 192.168.2.2 > log0 2>&1
ulric@test1:~/Git/pen$ taskset -c 1 ./pen -ddf 5001 192.168.2.3 > log1 2>&1

ulric@debtest:~$ iperf -c test1 -P 20
------------------------------------------------------------
Client connecting to test1, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 22] local 192.168.1.1 port 37658 connected with 192.168.1.2 port 5001
[ 12] local 192.168.1.1 port 37648 connected with 192.168.1.2 port 5001
[ 14] local 192.168.1.1 port 37650 connected with 192.168.1.2 port 5001
[ 9] local 192.168.1.1 port 37645 connected with 192.168.1.2 port 5001
[ 4] local 192.168.1.1 port 37640 connected with 192.168.1.2 port 5001
[ 13] local 192.168.1.1 port 37649 connected with 192.168.1.2 port 5001
[ 8] local 192.168.1.1 port 37644 connected with 192.168.1.2 port 5001
[ 5] local 192.168.1.1 port 37641 connected with 192.168.1.2 port 5001
[ 17] local 192.168.1.1 port 37653 connected with 192.168.1.2 port 5001
[ 11] local 192.168.1.1 port 37647 connected with 192.168.1.2 port 5001
[ 3] local 192.168.1.1 port 37639 connected with 192.168.1.2 port 5001
[ 15] local 192.168.1.1 port 37651 connected with 192.168.1.2 port 5001
[ 10] local 192.168.1.1 port 37646 connected with 192.168.1.2 port 5001
[ 7] local 192.168.1.1 port 37643 connected with 192.168.1.2 port 5001
[ 18] local 192.168.1.1 port 37654 connected with 192.168.1.2 port 5001
[ 6] local 192.168.1.1 port 37642 connected with 192.168.1.2 port 5001
[ 19] local 192.168.1.1 port 37655 connected with 192.168.1.2 port 5001
[ 16] local 192.168.1.1 port 37652 connected with 192.168.1.2 port 5001
[ 20] local 192.168.1.1 port 37656 connected with 192.168.1.2 port 5001
[ 21] local 192.168.1.1 port 37657 connected with 192.168.1.2 port 5001
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-10.0 sec 61.9 MBytes 51.9 Mbits/sec
[ 17] 0.0-10.0 sec 59.2 MBytes 49.7 Mbits/sec
[ 4] 0.0-10.0 sec 80.6 MBytes 67.6 Mbits/sec
[ 19] 0.0-10.0 sec 68.5 MBytes 57.4 Mbits/sec
[ 16] 0.0-10.0 sec 61.1 MBytes 51.2 Mbits/sec
[ 21] 0.0-10.0 sec 52.8 MBytes 44.2 Mbits/sec
[ 3] 0.0-10.0 sec 81.1 MBytes 67.8 Mbits/sec
[ 15] 0.0-10.0 sec 65.9 MBytes 55.1 Mbits/sec
[ 10] 0.0-10.0 sec 50.5 MBytes 42.2 Mbits/sec
[ 7] 0.0-10.0 sec 56.5 MBytes 47.2 Mbits/sec
[ 8] 0.0-10.0 sec 64.1 MBytes 53.6 Mbits/sec
[ 14] 0.0-10.1 sec 62.4 MBytes 51.9 Mbits/sec
[ 12] 0.0-10.1 sec 8.62 MBytes 7.17 Mbits/sec
[ 18] 0.0-10.1 sec 7.25 MBytes 6.02 Mbits/sec
[ 9] 0.0-10.1 sec 8.88 MBytes 7.36 Mbits/sec
[ 13] 0.0-10.1 sec 9.12 MBytes 7.56 Mbits/sec
[ 11] 0.0-10.1 sec 6.75 MBytes 5.60 Mbits/sec
[ 6] 0.0-10.1 sec 5.50 MBytes 4.56 Mbits/sec
[ 20] 0.0-10.1 sec 6.38 MBytes 5.29 Mbits/sec
[ 22] 0.0-10.1 sec 9.25 MBytes 7.66 Mbits/sec
[SUM] 0.0-10.1 sec 826 MBytes 684 Mbits/sec

One iperf server sees less traffic than the other:

[ 13] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 47869
[ 5] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 47874
[ 4] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 47871
[ 7] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 47876
[ 6] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 47875
[ 9] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 47883
[ 8] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 47881
[ 10] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 47884
[ 13] 0.0-10.1 sec 5.50 MBytes 4.56 Mbits/sec
[ 5] 0.0-10.1 sec 6.75 MBytes 5.60 Mbits/sec
[ 9] 0.0-10.1 sec 6.38 MBytes 5.28 Mbits/sec
[ 6] 0.0-10.1 sec 8.62 MBytes 7.14 Mbits/sec
[ 8] 0.0-10.1 sec 7.25 MBytes 6.01 Mbits/sec
[ 4] 0.0-10.1 sec 8.88 MBytes 7.34 Mbits/sec
[ 7] 0.0-10.1 sec 9.12 MBytes 7.55 Mbits/sec
[ 10] 0.0-10.1 sec 9.25 MBytes 7.65 Mbits/sec
[SUM] 0.0-10.1 sec 61.8 MBytes 51.0 Mbits/sec

and

[ 9] local 192.168.2.3 port 5001 connected with 192.168.2.1 port 59386
[ 10] local 192.168.2.3 port 5001 connected with 192.168.2.1 port 59388
[ 8] local 192.168.2.3 port 5001 connected with 192.168.2.1 port 59382
[ 7] local 192.168.2.3 port 5001 connected with 192.168.2.1 port 59381
[ 6] local 192.168.2.3 port 5001 connected with 192.168.2.1 port 59379
[ 5] local 192.168.2.3 port 5001 connected with 192.168.2.1 port 59377
[ 4] local 192.168.2.3 port 5001 connected with 192.168.2.1 port 59376
[ 15] local 192.168.2.3 port 5001 connected with 192.168.2.1 port 59375
[ 11] local 192.168.2.3 port 5001 connected with 192.168.2.1 port 59387
[ 12] local 192.168.2.3 port 5001 connected with 192.168.2.1 port 59394
[ 14] local 192.168.2.3 port 5001 connected with 192.168.2.1 port 59391
[ 13] local 192.168.2.3 port 5001 connected with 192.168.2.1 port 59389
[ 10] 0.0-10.1 sec 61.1 MBytes 51.0 Mbits/sec
[ 12] 0.0-10.1 sec 52.8 MBytes 44.0 Mbits/sec
[ 13] 0.0-10.1 sec 59.2 MBytes 49.4 Mbits/sec
[ 8] 0.0-10.1 sec 50.5 MBytes 42.0 Mbits/sec
[ 6] 0.0-10.1 sec 56.5 MBytes 47.0 Mbits/sec
[ 4] 0.0-10.1 sec 80.6 MBytes 67.1 Mbits/sec
[ 7] 0.0-10.1 sec 64.1 MBytes 53.3 Mbits/sec
[ 11] 0.0-10.1 sec 65.9 MBytes 54.7 Mbits/sec
[ 14] 0.0-10.1 sec 68.5 MBytes 57.0 Mbits/sec
[ 15] 0.0-10.1 sec 81.1 MBytes 67.4 Mbits/sec
[ 9] 0.0-10.1 sec 62.4 MBytes 51.8 Mbits/sec
[ 5] 0.0-10.1 sec 61.9 MBytes 51.4 Mbits/sec
[SUM] 0.0-10.1 sec 765 MBytes 635 Mbits/sec

Quite unbalanced:

ulric@test1:~/Git/pen$ wc -l log[01]
13711 log0
377374 log1
391085 total

Facebooktwittergoogle_plusredditpinterestlinkedinmail

Leave a Reply

Your email address will not be published. Required fields are marked *