CS526 S2012 Homework #4 Solution

Part 1. Clone cs526_LVSvApp for a set of four virtual machines.

Follow the "clone vApp" steps in http://cs.uccs.edu/~cs526/vmware/vapp.html#clonevapp to clone a <login>_LVSvApp from cs526_LVSvApp in your folder. Make sure to substitute easguest with your loginname. Go direct to clone vapp section. For hw4, skip the create vapp and populate vapp steps unless you would like to practice the creation of vApp template yourself.
Report what hosts are allocated to serve these four vms.

Ans: Click on the <login>LVSvApp. The host assignments for 4 vms will show in the host column.

host assignment

cs526_lvs is allocated to eas-vsphere01.uccs.edu
cs526_rs1 is allocated to eas-vsphere07.uccs.edu
cs526_rs2A is allocated to eas-vsphere04.uccs.edu
cs526_rs3A is allocated to eas-vsphere02.uccs.edu

In some case, the host entry is not shown or "unknown".

unknown host assignmnet

You can click on the name of the vm and then check on the monitor tab and related event. You will find to which host the vm is allocated.
find assigned host

cs526_lvs is allocated to eas-vsphere01.uccs.edu
cs526_rs1 is allocated to eas-vsphere01.uccs.edu
cs526_rs2A is allocated to eas-vsphere06.uccs.edu
cs526_rs3A is allocated to eas-vsphere05.uccs.edu

Try migrating rs1 to other host such as eas-vsphere02 so that the cluster's performance can be better.

migrate vm

change host

select host to migrate

select resource pool

I should select cs526_LVSvApp, by selecting CS5260. The system migrates rs1 to cs526 outside of cs526_LVSvApp.

I need to drag it back to cs526_LVSvApp later.

select priority of migration

rs1 migrate to vsphere02

Now we have

cs526_lvs is allocated to eas-vsphere01.uccs.edu
cs526_rs1 is allocated to eas-vsphere02.uccs.edu
cs526_rs2A is allocated to eas-vsphere06.uccs.edu
cs526_rs3A is allocated to eas-vsphere05.uccs.edu

Note that part 1 does not requrie the migration to spread the vms on totally different hosts.

You may consider that if migration opition is available and you have ways to correctly estimate/predict the load statue of hosts for the future.

Part 2. Setup a cluster based on Apache Load Balancer.

Through the consoles of the virutal machines configure the these four virtual machines with your designated IP addresses. Modify the default web page to indicate the real server name and your loginname.

Read ubuntu apache setup and follow sample 3 of http://cs.uccs.edu/~cs526/apacheProxyBalancer.html to set up a cluster using lvs as a balancer and rs1, rs2A, rs3A as real servers in your LVSvApp. Note that there is 000-chowlb sites in /etc/apache2/sites-enabled that you need to modify for your load balancer cluster.

Ans: cp /etc/apache2/sites-available/default /etc/apache2/sites-available/chowlb Add the following lines before </VirtualHost> of chowlb file.

ProxyPass / balancer://mycluster/
<Proxy balancer://mycluster>
BalancerMember http://chowrs1.csnet.uccs.edu loadfactor=1
BalancerMember http://chowrs2.csnet.uccs.edu loadfactor=1
BalancerMember http://chowrs3.csnet.uccs.edu loadfactor=1

ProxySet lbmethod=bybusyness
</Proxy>

run "a2ensite chowlb"

root@chowdip:/etc/apache2/sites-available# a2ensite chowlb
Enabling site chowlb.
To activate the new configuration, you need to run:
  service apache2 reload
root@chowdip:/etc/apache2/sites-available# service apache2 reload
Syntax error on line 41 of /etc/apache2/sites-enabled/chowlb:
Invalid command 'ProxyPass', perhaps misspelled or defined by a module not included in the server configuration
Action 'configtest' failed.
The Apache error log may have more information.
   ...fail!
root@chowdip:/etc/apache2/sites-available# a2nemod proxy_balancer
No command 'a2nemod' found, did you mean:
 Command 'a2enmod' from package 'apache2.2-common' (main)
a2nemod: command not found
root@chowdip:/etc/apache2/sites-available# a2enmod proxy_balancer
Considering dependency proxy for proxy_balancer:
Enabling module proxy.
Enabling module proxy_balancer.
To activate the new configuration, you need to run:
  service apache2 restart
root@chowdip:/etc/apache2/sites-available# a2enmod proxy_http
Considering dependency proxy for proxy_http:
Module proxy already enabled
Enabling module proxy_http.
To activate the new configuration, you need to run:
  service apache2 restart
root@chowdip:/etc/apache2/sites-available# service apache2 restart
 * Restarting web server apache2
 ... waiting    ...done.
root@chowdip:/etc/apache2/sites-available#

Set the load balancing algorithm to be Pending Request Counting.
Use ab to benchmark those three real web servers. Set their loadfactor number accordingly on the proxy for the best performance.
Try to evaluate the cluster performance usig ab -c 20 -n 5000 http://<login>dip.csnet.uccs.edu. Discuss the observed performance.

Ans: The following table shows the ab benchmark results on rs1, rs2, rs3, dip, load balancing (dipLB), and load balancing (after double memory size and triple the vcpu). Note that apache load balancing module only take load factor value between 1-100. 147 and 146 were not accepted. 74, 73, 50 were used in this exercise. It seems that the aggregated throughput for the load balancer is not much bigger than that of the fastest backend real server.
Note that the /var/www/ contains a benchmark directory. You can use it or create your own benchmark web pages.
Make sure you power off the vApp after each session. They can be power off as a unit.
After you finish the part2, you may want to consider remove 000-chowlb or even 001-default as <VirtualHost *:80> may catch requests for your vip.

Part 3. Setup a LVS DR cluster.

For each virtual machine in the LVS DR cluster,

cd /home/csnet/lvs and modify the script for the related role of the virutal machine by changing the vip, rip, dip numbers.

Ans: The top of rs_dr.sh for rs1 is changed to

gwip="128.198.49.65"
rip="128.198.49.117"
vip="128.198.49.120"

similar for rs2

gwip="128.198.49.65"
rip="128.198.49.119"
vip="128.198.49.120"

and for rs3

gwip="128.198.49.65"
rip="128.198.49.118"
vip="128.198.49.120"

The top of lvs_dr.sh for lvs is changed to

vipno="115"
vip="128.198.49.120"
dip="128.198.49.121"
rs1="128.198.49.117"
rs2="128.198.49.118"
rs3="128.198.49.119"


#add http to VIP with round robin scheduling
/sbin/ipvsadm -A -t $vip:http -s wrr
/sbin/ipvsadm -A -t $vip:ssh -s wlc

#forward http to realserver using direct routing with weight 1
/sbin/ipvsadm -a -t $vip:http -r $rs1 -g -w 74
/sbin/ipvsadm -a -t $vip:http -r $rs2 -g -w 73
/sbin/ipvsadm -a -t $vip:http -r $rs3 -g -w 50

#forward ssh to realserver using direct routing with weight 1
/sbin/ipvsadm -a -t $vip:ssh -r $rs1 -g -w 74
/sbin/ipvsadm -a -t $vip:ssh -r $rs2 -g -w 73
/sbin/ipvsadm -a -t $vip:ssh -r $rs3 -g -w 50

Run the scripts to set up the LVS DR cluster (on lvs run lvs_dr.sh; on real servers, run rs_dr.sh).

On lvs, decide what is a good schedule algorithm and weights for real servers to use for achieving the best web service performance. Same for the ssl terminal service.
Ans: It is decided to use wrr for web services and wlc for ssh terminal service.
Generate 7 SSL terminal sessions. Capture the terminal session data on SSH Secure Shell Client or Putty that indicate which real servers are assigned to serve the requests. Briefly describe the pattern.
Ans: See screendumps of the seven SSH Secure Shell Sessions to cs526vip.csnet.uccs.edu.
The pattern for allocating real server is rs3, rs2, rs1, rs1, rs2, rs3, rs1.

Capture 7 web access sessions to the <login>vip.csnet.uccs.edu using lynx (a text based browser) on gandalf. Briefly describe the pattern.

Ans:

    Lynx session 1:



[head4.jpg] 


    This is cs526rs1.csnet.uccs.edu for CS526, created by chow. It works!
Lynx Session 2:

                                 [head4.jpg]

    This is cs526rs2.csnet.uccs.edu for CS526, created by chow. It works!

Lynx Session 3: 
                                 [head4.jpg]

    This is cs526rs1.csnet.uccs.edu for CS526, created by chow. It works!

Lynx Session 4:
                                 [head4.jpg]

    This is cs526rs2.csnet.uccs.edu for CS526, created by chow. It works!

Lynx Session 5:
                                 [head4.jpg]

    This is cs526rs1.csnet.uccs.edu for CS526, created by chow. It works!

Lynx Session 6:
                                 [head4.jpg]

    This is cs526rs2.csnet.uccs.edu for CS526, created by chow. It works!

Lynx Session 7:
                                 [head4.jpg]

    This is cs526rs1.csnet.uccs.edu for CS526, created by chow. It works!

Therefore the pattern is rs1, rs2, rs1, rs2, rs1, rs2, rs1, rs2, rs1, rs2, rs1, ..
rs3 was never seen.  ipvsadm status also confirms that wrr assigned real servers different than wlc.


root@cs526dip:~/lvs# ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  128.198.49.120:ssh wlc
  -> 128.198.49.117:ssh           Route   74     0          0         
  -> 128.198.49.118:ssh           Route   73     0          0         
  -> 128.198.49.119:ssh           Route   50     0          0         
TCP  128.198.49.120:www wrr
  -> 128.198.49.117:www           Route   74     0          15        
  -> 128.198.49.118:www           Route   73     0          16        
  -> 128.198.49.119:www           Route   50     0          0

Save the above SSL/Web sessions results as .html files and reference them in your hw4.html.
Try to evaluate the cluster performance using ab -c 20 -n 5000 http://<login>vip.csnet.uccs.edu. Discuss the observed performance and compared that with part 2.
Make sure you power off the vApp after each session.
Ans: For the large benchmark web page. We get the following results. The lvs was surprising slow and does not achieve any aggregated performance from individual servers. It could be the transient behavior.

For small web page (index.html), we got the following results:

It seems to achieve the aggregated request throughput, but there are failed requests. Compared with apache load balancer, it seems to perform much better.

Questions.

Besides the potential performance enhancement (debatable based on your observed performance at part 2), what are other advantages we should use apache load balancer?
Ans: It does not require any changes on the real server and it allows adjusting of weights.

What are the observed LVS DR cluster performance and what performance you expected it to achieve? Provide an explanation on the difference.

Ans: The two sets of runs with different web page sizes. Yield dramatically different results. One of experiments with all vms including the testing machines allocated to the same hosts seem to yield a consistent result for both page sizes. One explanation is the network traffice and processing overhead could be drastically different. It could have significant impact on the performance.
Here is the results on nisssc cluster:

      
csnet@chowClient:~/benchmark$ ab -c 20 -n 5000 http://128.198.50.166/  (RS1)
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 128.198.50.166 (be patient)
Completed 500 requests
Completed 1000 requests
Completed 1500 requests
Completed 2000 requests
Completed 2500 requests
Completed 3000 requests
Completed 3500 requests
Completed 4000 requests
Completed 4500 requests
Completed 5000 requests
Finished 5000 requests


Server Software:        Apache/2.2.16
Server Hostname:        128.198.50.166
Server Port:            80

Document Path:          /
Document Length:        507 bytes

Concurrency Level:      20
Time taken for tests:   1.776 seconds
Complete requests:      5000
Failed requests:        0
Write errors:           0
Total transferred:      3915000 bytes
HTML transferred:       2535000 bytes
Requests per second:    2815.14 [#/sec] (mean)
Time per request:       7.104 [ms] (mean)
Time per request:       0.355 [ms] (mean, across all concurrent requests)
Transfer rate:          2152.59 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       6
Processing:     0    1   0.1      1       2
Waiting:        0    1   0.1      1       2
Total:          1    1   0.2      1       7

Percentage of the requests served within a certain time (ms)
  50%      1
  66%      1
  75%      1
  80%      1
  90%      1
  95%      1
  98%      1
  99%      1
 100%      7 (longest request)
csnet@chowClient:~/benchmark$ ab -c 20 -n 5000 http://128.198.50.167/ (RS2)
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 128.198.50.167 (be patient)
Completed 500 requests
Completed 1000 requests
Completed 1500 requests
Completed 2000 requests
Completed 2500 requests
Completed 3000 requests
Completed 3500 requests
Completed 4000 requests
Completed 4500 requests
Completed 5000 requests
Finished 5000 requests


Server Software:        Apache/2.2.16
Server Hostname:        128.198.50.167
Server Port:            80

Document Path:          /
Document Length:        508 bytes

Concurrency Level:      20
Time taken for tests:   1.808 seconds
Complete requests:      5000
Failed requests:        0
Write errors:           0
Total transferred:      3920000 bytes
HTML transferred:       2540000 bytes
Requests per second:    2765.69 [#/sec] (mean)
Time per request:       7.231 [ms] (mean)
Time per request:       0.362 [ms] (mean, across all concurrent requests)
Transfer rate:          2117.48 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       7
Processing:     0    1   1.1      1      44
Waiting:        0    1   1.1      1      44
Total:          1    1   1.1      1      44

Percentage of the requests served within a certain time (ms)
  50%      1
  66%      1
  75%      1
  80%      1
  90%      1
  95%      1
  98%      1
  99%      1
 100%     44 (longest request)
csnet@chowClient:~/benchmark$ ab -c 20 -n 5000 http://128.198.50.168/ (RS3)
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 128.198.50.168 (be patient)
Completed 500 requests
Completed 1000 requests
Completed 1500 requests
Completed 2000 requests
Completed 2500 requests
Completed 3000 requests
Completed 3500 requests
Completed 4000 requests
Completed 4500 requests
Completed 5000 requests
Finished 5000 requests


Server Software:        Apache/2.2.16
Server Hostname:        128.198.50.168
Server Port:            80

Document Path:          /
Document Length:        507 bytes

Concurrency Level:      20
Time taken for tests:   1.801 seconds
Complete requests:      5000
Failed requests:        0
Write errors:           0
Total transferred:      3915000 bytes
HTML transferred:       2535000 bytes
Requests per second:    2776.02 [#/sec] (mean)
Time per request:       7.205 [ms] (mean)
Time per request:       0.360 [ms] (mean, across all concurrent requests)
Transfer rate:          2122.68 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       4
Processing:     0    1   0.1      1       3
Waiting:        0    1   0.1      1       3
Total:          1    1   0.2      1       5

Percentage of the requests served within a certain time (ms)
  50%      1
  66%      1
  75%      1
  80%      1
  90%      1
  95%      1
  98%      1
  99%      1
 100%      5 (longest request)
csnet@chowClient:~/benchmark$ ab -c 20 -n 5000 http://128.198.50.190/ (VIP)
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 128.198.50.190 (be patient)
Completed 500 requests
Completed 1000 requests
Completed 1500 requests
Completed 2000 requests
Completed 2500 requests
Completed 3000 requests
Completed 3500 requests
Completed 4000 requests
Completed 4500 requests
Completed 5000 requests
Finished 5000 requests


Server Software:        Apache/2.2.16
Server Hostname:        128.198.50.190
Server Port:            80

Document Path:          /
Document Length:        507 bytes

Concurrency Level:      20
Time taken for tests:   0.543 seconds
Complete requests:      5000
Failed requests:        1667  (same situation is observed in our LVS-vApp for small page size)
   (Connect: 0, Receive: 0, Length: 1667, Exceptions: 0)
Write errors:           0
Total transferred:      3916667 bytes
HTML transferred:       2536667 bytes
Requests per second:    9216.49 [#/sec] (mean)
Time per request:       2.170 [ms] (mean)
Time per request:       0.109 [ms] (mean, across all concurrent requests)
Transfer rate:          7050.37 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   0.3      1       3
Processing:     0    2   1.0      1       8
Waiting:        0    1   1.0      1       8
Total:          0    2   1.1      2       9

Percentage of the requests served within a certain time (ms)
  50%      2
  66%      2
  75%      3
  80%      3
  90%      4
  95%      4
  98%      4
  99%      6
 100%      9 (longest request)
csnet@chowClient:~/benchmark$

Given the availability of these two techniques, provide one example where you will definitely use apache load balancer and another example where you will definitely use LVS DR.
Ans: Example 1: a cluster consists of java servlets. Apache load balancer will be much easier to configure and LVS DR is not good at care of sticky connections. Example 2: a cluster of SSH terminal servers using idle PCs in the lab. Definitely use LVS DR.
From Part 2 to Part 3, while not disable apache load balancer, LVS-DR will still work (even apache load balancer is running). Explain why. Hint: http://cs.uccs.edu/~cs526/lvs/lvs.pptx#9. Note that as a good practice, you should disable and shutdown those server processes that are no longer needed.
Ans: LVS-DR intercepts the http request at kernel level and therefore those requests were never sent above to the apache server which is running at user's space.

Create a homework web page hw4.html in your cs526 personal web page directory. Email the url and anawers of the above questions to cchow@uccs.edu and aalmuray@uccs.edu with subject field "CS526 hw4".

Solution to Homework #4. Load Balancing a Cluster Using Apache Proxy Balancer and LVS DR