Goal:

• Learn to use VMWare virtual machines served by vmware server
• Learn Ubuntu OS and related commands for setting up network interface and apache web page.
• Learn how to configure/test LVS DR cluster.
• Learn the tradeoff among LVS NAT, Tunnel, and DR configurations.

Assignment Date: 3/11/2009
Due Day: Answer of Questions 3/16/2009 to dropbox; The hw3.html with exercise results 3/31/2009.
Related documents:

• http://www.linux-mag.com/launchpad/business-class-hpc/main/1501 (require to setup account with linux magzine).
  local copy http://cs.uccs.edu/~cs526/lvs/www.linux-mag.com-index2.pdf
• http://www.linuxvirtualserver.org/VS-DRouting.html
• http://www.austintek.com/LVS/LVS-HOWTO/
• ipvsadm man page. login to any of the cluster node with ipvsadm installed. "man ipvsadm".

Description:

Logistics:

• We have set up 4 machines, sis.csnet.uccs.edu, walden, walrus, wind as VMware host machines. I have created a folder with your login name in d:\work\cs526\S2009. Inside that directory, a lvs_chow folder with four virtual machines (lvs, rs1, rs2, rs3) is prepared for you. You can select "Add Virtual Machine to Inventory" to add these virtual machines your host machine based on the VMware Infrastructure web based interface. Then start and configure them using a set of shell scripts to set up lvs DR type cluster.

• Name	VIP	Assigned IP Addresses	DNS names	Assigned VMWare Host
  Archer, Nathan James	128.198.61.15	128.198.61.37-38, 16-17	narcher; narcherx	sis.csnet.uccs.edu
  Ardrey, Brian Jeffrey		128.198.61.39-40	bardrey; bardreyx	sis.csnet.uccs.edu
  Asanghanwa, Eustace Ngwa	128.198.61.116	128.198.61.41-42	easangha; easanghax	sis.csnet.uccs.edu
  Bates, Andrew Leonard		128.198.61.43-44	abates; abatesx	sis.csnet.uccs.edu
  Beckmann, Charles Robert		128.198.61.45-46	cbeckma2; cbeckma2x	sis.csnet.uccs.edu
  Champlin, William Patrick	128.198.61.122	128.198.61.47-48, 123-124	wchampli; wchamplix	walden.csnet.uccs.edu
  Cohen, Michael Javan		128.198.61.49-50	mcohen; mcohenx	walden.csnet.uccs.edu
  Crowther, Brandon Glade	128.198.60.100	128.198.60. 101-104	bcrowthe; bcrowthex	walden.csnet.uccs.edu
  Fu, Guozhi	128.198.61.26	128.198.61.53-54, 27-28	gfu; gfux	walden.csnet.uccs.edu
  Gonzalez, Luis Jose	128.198.61.18	128.198.61.55-56, 19-20	lgonzal6; lgonzal6x	walden.csnet.uccs.edu
  Hinson, Jeffrey Scott	128.198.60.110	128.198.60.111-4	jhinson; jhinsonx	walrus.csnet.uccs.edu
  Huynh, Philip Huu	128.198.61.25	128.198.61.59-60	phuynh; phuynhx	walrus.csnet.uccs.edu
  Iyer, Vijay Navin	128.198.61.117	128.198.61.61-62	viyer; viyerx	walrus.csnet.uccs.edu
  Jahnke, Shane Rulon	128.198.61.116	128.198.61.57-58	sjahnke; sjahnkex	walrus.csnet.uccs.edu
  Johnson, Raoul Christopher	128.198.61.118	128.198.61.65-66	rjohnso9; rjohnso9x	walrus.csnet.uccs.edu
  Karikatti, Smita	128.198.61.120	128.198.61.67-68	skarikat; skarikatx	wind.csnet.uccs.edu
  Karikattisiddappa, Shambuling	128.198.61.120	128.198.61.69-70	skarika2; skarika2x	wind.csnet.uccs.edu
  Lama, Palden		128.198.61.71-72	plama; plamax	wind.csnet.uccs.edu
  Logan, James Phillip	128.198.61.115	128.198.61.73-74	jlogan; jloganx	wind.csnet.uccs.edu
  Magee, Anthony William	128.198.61.117	128.198.61.75-76	amagee; amageex	wind.csnet.uccs.edu
  Miller, Michael Scott	128.198.60.120	128.198.60.121-124	mmiller9; mmiller9x	walrus.csnet.uccs.edu
  Mohsen, Fadi Fuad	128.198.61.25	128.198.61.79-80	fmohsen; fmohsenx	sis.csnet.uccs.edu
  Morris, John Matthew	128.198.61.119	128.198.61.81-82	jmorris4; jmorris4x	sis.csnet.uccs.edu
  Niehoff, Nicholas William	128.198.61.118	128.198.61.83-84	nniehoff; nniehoffx	sis.csnet.uccs.edu
  Poley, Timothy William		128.198.61.85-86	tpoley; tpoleyx	sis.csnet.uccs.edu
  Reddy, Sreedeepti		128.198.61.87-88	sreddy; sreddyx	walden.csnet.uccs.edu
  Ruiz, Daniel Alfonso	128.198.61.29	128.198.61.89-90, 30-31	druiz; druizx	walden.csnet.uccs.edu
  Sanchez-vasquez, Phillip Troy	128.198.61.32	128.198.61.91-92, 33-34	pgurule; pgurulex	walden.csnet.uccs.edu
  Sapkota, Archana		128.198.61.93-94	asapkota; asapkotax	walden.csnet.uccs.edu
  Shuster, Christopher Michael		128.198.61.95-96	cshuster; cshusterx	walden.csnet.uccs.edu
  Sterling, Nicholas Van		128.198.61.97-98	nsterlin; nsterlinx	walrus.csnet.uccs.edu
  Taylor, Joe Bert	128.198.61.12	128.198.61.99-100, 13-14	jtaylor8; jtaylor8x	walrus.csnet.uccs.edu
  Thomas, Ryan Matthew	128.198.61.121	128.198.61.101-102	rthomas2; rthomas2x	walrus.csnet.uccs.edu
  Thorpe, Brian Steven	128.198.61.119	128.198.61.103-104	bthorpe; bthorpex	walrus.csnet.uccs.edu
  Torres, Peter	128.198.61.2	128.198.61.105-106, 3-4	ptorres; ptorresx	walrus.csnet.uccs.edu
  Tran, Kim-lan Lim	128.198.61.121	128.198.61.107-108	ktran; ktranx	wind.csnet.uccs.edu
  Turner, George Eugene		128.198.61.109-110	gturner; gturnerx	wind.csnet.uccs.edu
  Wilke, Brian Gregory		128.198.61.111-112	bwilke; bwilkex	wind.csnet.uccs.edu
  Wilson, Jimmy Leon	128.198.61.115	128.198.61.113-114	jwilson5; jwilson5x	wind.csnet.uccs.edu

• You can work alone or with one of your classmates as two person teams. Due to the limited resources and memory (4GB), you are encouraged to form two person teams. Note that you need to 5 IP addresses for this exercise. 4 for lvs director, real servers rs1-3, and one VIP. Email me your team composition and I will assign vip for you, or two additonal IP addresses if work alone.

Part 1. Construct LVS-DR cluster.

• You access vmware server by typing http://walden.ucs.edu:8222/ or https://walden.uccs.edu:8333/ if walden is your designated host machine. Note that These vmware hosts can only be accessed through the uccsvpn.
• When you first access with https, you will be presented with server certificate but since the certificate is self-signed without being signed known commercial Certificate Authority (CA), you will be warned about the potential threat. Choose to accept the server certificate. For firefox, click "Add Exception"
  
  
  Click "Get Certificate" then hit "Confirm Security Exception" button.
  
• You will be prompt for selecting your client certificate just click ok.
  
• Note that it will automatically switch to https connection for login/password secure access. You will be asked to enter your login and password. Type your ufp account name as login and "#A" followed by the SID with dash nnn-nn-nnnn as password.
  
• You will see the remove console web page appear.
  
• Select "Virtual Machine" menu and "Add Virtual Machine to Inventory" menuitem. Chose the cs526 datastore and your folder. In this case we will choose the wchampli/lvs_chow/rs1/Ubuntu.vmx file. Double click on the file and click OK. And it will be added to the Inventory panel on the left side.
  
• Click on the vm label on Inventory panel with <login>_lvs1rs1. Hit the play botton or "Power On" command on the right panel. Select the Console tab to see the virtual console screen.
  
• Note that if this is your first time use of the interface. You will be asked to install vmware remote console plug-in.
  
• Click "Install plug-in". After the installation, if you hit power on button again, you should see the following
• Click "I _copied it" choice. This will generate unique UUID and ethernet MAC address. Therefore the system will not be confused with the original copy of the virtual machine, or other potential 38 copies of them. Click OK.
  
• 
• Click inside the window. It will start the remote console as a separte flowing window.
  
• Hit enter to get the ubuntu login prompt as shown above. Note that the original OS login prompt will be hidden among the server starting msgs.
• The default login is csnet. The password will be announced in class.
• To switch to root user shell, type "sudo bash" and type the password again.
• Note that to allow the console window to accept your keyboard input, you need to move the mouse inside the window area and click the mouse button. The console will grab on your keyboard and mouse input. To release the grab, hit control-alt to get out of the window.

Setup real server 1 (rs1):

• Login to the real server with "csnet" as login and its password.

• First thing to do is to set up the rs1 with the right IP address.
  • Assume that Bill has five IP addresses and he assigns 128.198.61.48 for the rs1.
  • To assign an IP address to an ethernet interface, we use "ifconfig ethn 128.198.61.48" where ethn is the interface device id. Our virtual machines all have two ethernet interface. The first one is of bridged type which mapped to the physical ethernet interface of the host machine and typicall associated with an external IP address. The second one is host-only type which allows us to use Private LAN address. In LVS DR configuration, we are not using the 2nd interface, one interface is enough. If you would like to try LVS NAT or DMZ firewall configuration, you can use the 2nd interface.
  • If you try the "ifconfig eth0 128.198.61.48", you may be surprised to see the system replies with
    

    root@lvs1rs1:~# ifconfig eth0 128.198.61.49
    SIOCSIFADDR: No such device
    eth0: ERROR while getting interface flags: No such device
    

  • This is due to Guest Ubuntu Linux system still remember the old MAC address in the original virtual machine. When we indicates this is a copy of a virtual machine, the vmware server changed the two MAC addresses. When the Guest OS boots, it detects two new MAC address and thought that the machine was plugged in with two new additional cards and assigns them with new device ids. It also recognizes the two old network interface is no longer active but they still keep their old, lower number device ids.
• To find out what interfaces areactive, type "ifconfig -a"

  root@lvs1rs1:~# ifconfig -a
  eth8      Link encap:Ethernet  HWaddr 00:0C:29:67:88:D6  
  
            BROADCAST MULTICAST  MTU:1500  Metric:1
            RX packets:0 errors:0 dropped:0 overruns:0 frame:0
            TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:1000 
            RX bytes:0 (0.0 b)  TX bytes:594 (594.0 b)
            Interrupt:17 Base address:0x1480 
  
  eth9      Link encap:Ethernet  HWaddr 00:0C:29:67:88:CC  
            inet6 addr: fe80::20c:29ff:fe67:88cc/64 Scope:Link
  
            RX packets:132 errors:0 dropped:0 overruns:0 frame:0
            TX packets:91 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:1000 
            RX bytes:16497 (16.1 KB)  TX bytes:10221 (9.9 KB)
            Interrupt:16 Base address:0x1400 
  
  lo        Link encap:Local Loopback  
            inet addr:127.0.0.1  Mask:255.0.0.0
            inet6 addr: ::1/128 Scope:Host
            UP LOOPBACK RUNNING  MTU:16436  Metric:1
            RX packets:3 errors:0 dropped:0 overruns:0 frame:0
            TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:0 
            RX bytes:336 (336.0 b)  TX bytes:336 (336.0 b)
  
          

• This shows we have two active interfaces. But if you think eth8 is mapped to the Network Adapter 1, you are wrong, actually it is eth9. How do I find that out?
  • Question 3.1: Explain how do I find out eth8 is not mapped to Network Adapter 1.
  • Question 3.2: Explain why eth9 instead of eth8 is mapped to network adapter.
  • Question 3.3. What is NAT type vmware network interface? What is host-only type vmware interface? Give one configure example on how you will use each type of network interfaces.
  • Question 3.4: At least how many times this virtual machine is copied, since its original installation?
• To allow outside access to this virtual machine, we will configure eth9 with the following command:
  ifconfig eth9 128.198.61.48 netmask 255.255.255.128
  route add default gw 128.198.61.1
  
  Then I "ping 128.198.1.250" to verify that indeed I get a response from outside (our uccs dns server).
• The console window does not capture the command/response. I would suggest you login usign SSH shell client to your virtual machine after setting up the network interface. You can copy and paste screen image much easier.
• In csnet's lvs directory, you will see a few lvs real server setup scripts:

  root@lvs1rs1:~# cd lvs
  root@lvs1rs1:~/lvs# ls
  images  index.html  reset_rs_dr.sh  reset_rs_nat.sh  rs_dr.sh  rs_nat.sh
  
    

• You can use rs_dr.sh to set up the real server rs1. But you need to modify the related DIP, VIP and gateway address. For example, 128.198.61.141 should be 128.198.61.48, while the gateway should be 128.198.61.1
• Teplace 128.198.61.131 (VIP) with the one I assigned for you.
• Rdit /etc/sysctl.conf
  

  by replacing
  
  # just edit /etc/sysctl.conf and add
  # net.ipv4.conf.eth7.arp_ignore = 1
  # net.ipv4.conf.eth7.arp_announce = 2
  
  
  with
  
  # just edit /etc/sysctl.conf and add
  # net.ipv4.conf.eth9.arp_ignore = 1
  # net.ipv4.conf.eth9.arp_announce = 2
  
        

• Setup the default web page with the real server domain name, your name by modifying the index.html in /var
  • Replace
    This is lvs1rs1.csnet.uccs.edu for CS526. It works!
    wtih
    This is <login>_lvs1rs1.csnet.uccs.edu for CS526 by <your name>. It works!
  • You can change the color by replacing
    background-image: url("images/redgbg.jpg");
    with a different image file in images directory.
    
• verify the change by typing the url, say http://128.198.61.48/ replace 128.198.61.48 with your own rip1.
• Benchmark real server web performance. Use ab program from viva.ucs.edu to http://<your rip>. Report the requests/sec performance number. We will use that to estimate the weights when configuring the http service on director.
• Repeat the same process to configure real server rs2 and rs3. But select the ubuntu.vmx file in lvs_chow/rs2 and lvs_chow/rs3 respectively, and use the right DIP of your choice.

Setup LVS DR director:

• setup the LVS DR director virtual machine by selecting the ubuntu.vmx file in lvs_chow/lvs as above.
• login
• cd lvs
• modify and run the lvs_dr.sh
  • replace all instances of 128.198.61.51 with your own VIP
  • replace the real sever IP addresses with the IP addresses of your designated real servers.
  • In ipvsadm -A commands, configure -wrr for web service and -wlc for ssh service.
  • in each of the ipvsadm -a command, replace the -w n weight number with your estimated weight relative to the peformance of the three real servers you have. The bigger the number the faster the server and they will be allocated with more requests by the director.
  • ./lvs_dr.sh

Test the LVS-DR cluster.

• Test http web access:
  • Start three web browsers (due to caching, we may want to use different web browsers, e.g., IE7, firefox, and opera, so that we can access the web pages of all three real servers.
  • Make sure you use the same url http://<vip ip address>/ on your web browser.
  • Use shift-reload to see if a new web page is retrieved. You can also wait a minute if the same is retrieve
  • Observe the pattern on these web access.
  • Run ipvsadm and capture the content of ipvsadm status table. Watch the active and inactive connection numbers.
  • Explain how the wrr scheduling and the weight assignment matches how requests are distributed.
  • Report the observation, the ipvsadm status, and explanation.
• Test ssh service:
  • Create around 10 SSH shell terminal instances and access the same vip address.
  • Observe what real sever is allocated to serve those 10 SSH requests.
  • Run ipvsadm and capture the content of ipvsadm status table.
  • Explain how the wlc scheduling and the weight assignment matches how the 10 requests are distributed.
  • Report the observation, the ipvsadm status, and explanation.
• Benchmark the LVS-Dr cluster.
  • Use ab to measure the performance of the cluster by accessing http://<vip>
  • Adjust the weight or scheduling disciplines and see if you can improve the perforfmance.
  • Report your finding and the benchmark results.

Summarize the above report as a web page hw3.html in your cs526 personal web page. Submitt the url.

Part 2.

• Question 3.5: In what cases wlc perform better than wrr? Describe a simple case that highlights that.
• Question 3.6: Why LVS-NAT will be slower than LVS-DR?
• Question 3.7: When LVS-Tunnel will be performance better LVS-DR? What additional information director need to know to allocate requests to a "better" real server?

Q&A:

1. Q: The machine which I choose to run director does not have ipvsadm installed. It suggests that I put cdrom with the origional ubuntu os, but remotely I can not access the machine? What should I do?
   
   
   Ans:
   

   You probably run the ipvsadm and got the friendly msg from ubuntu:

   csnet@r107:~$ sudo bash
   [sudo] password for csnet:
   root@r107:~# ipvsadm
   The program 'ipvsadm' is currently not installed.  You can install it by typing:
   apt-get install ipvsadm
   bash: ipvsadm: command not found
   root@r107:~# apt-get install ipvsadm
   Reading package lists... Done
   Building dependency tree      
   Reading state information... Done
   Suggested packages:
     heartbeat keepalived ldirectord
   The following NEW packages will be installed:
     ipvsadm
   0 upgraded, 1 newly installed, 0 to remove and 29 not upgraded.
   Need to get 0B/47.6kB of archives.
   After unpacking 242kB of additional disk space will be used.
   Media change: please insert the disc labeled
    'Ubuntu-Server 7.10 _Gutsy Gibbon_ - Release i386 (20071016)'
   in the drive '/cdrom/' and press enter

   Unfortunately, remotely we cannot load the cdrom which contain a copy of related software package.
   To tell apt-get not to search the software package on original cdrom and search the mirror site instead, try the following

   • sudo vi /etc/apt/sources.list
     sources.list indicates the order of the search list for software packages.
   • Comment out  line 4 by putting # in front of the line, as follows:
     #deb cdrom:[Ubuntu-Server 7.10 _Gutsy Gibbon_ - Release i386 (20071016)]/ gutsy main restricted
   • sudo apt-get install ipvsadm

   Now we get
   root@r107:~# apt-get install ipvsadm
   Reading package lists... Done
   Building dependency tree      
   Reading state information... Done
   Suggested packages:
     heartbeat keepalived ldirectord
   The following NEW packages will be installed:
     ipvsadm
   0 upgraded, 1 newly installed, 0 to remove and 29 not upgraded.
   Need to get 47.6kB of archives.
   After unpacking 242kB of additional disk space will be used.
   Get:1 http://us.archive.ubuntu.com gutsy/main ipvsadm 1.24+1.21-1.4ubuntu1 [47.6kB]
   Fetched 47.6kB in 0s (62.1kB/s)
   Preconfiguring packages ...
   Selecting previously deselected package ipvsadm.
   (Reading database ... 17971 files and directories currently installed.)
   Unpacking ipvsadm (from .../ipvsadm_1.24+1.21-1.4ubuntu1_i386.deb) ...
   Setting up ipvsadm (1.24+1.21-1.4ubuntu1) ...

   root@r107:~# ipvsadm
   IP Virtual Server version 1.2.1 (size=4096)
   Prot LocalAddress:Port Scheduler Flags
     -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
   root@r107:~#

   
   
2.