Goal:

• Learn how to use the Cloud Computing services provided by Amazon Web Services to create and manage the virtual machines through the web brower and the command line API.
• Evaluate the performance of virtual machines running at data centers on different regions (current there are eight regions: US - N. Virginia, US - N. California, US - Portland, EU - Ireland, APAC - Singapore, APAC - Tokyo, Sounth America, Sao Paulo)
• Learn the cost and perfrormance trade-off of using public cloud and private cloud through a simple capacity planning exercise.

Assignment Date: 4/18/2012
Due Day: Parts 1&2 due 4/25/2012; Part 3 due 4/30/2012.
Related documents:

• http://cs.uccs.edu/~cs526/studentproj/projS2010/mnamburu/doc/CS526 Project Report.pdf
• http://docs.amazonwebservices.com/AWSEC2/latest/GettingStartedGuide/
• http://docs.amazonwebservices.com/AWSEC2/latest/CommandLineReference/
• http://docs.amazonwebservices.com/AWSEC2/latest/APIReference/

Description:

Logistics:

• I have used AWS IAM GUI to create your account (IAM users) using the same ufp login and same password for walrus (#a followed by SID without dash).
• The secure web site for accessing the AWS management console will be announced in class. hint: https://xxxxxx.signin.aws.amazon.com/console/

Part 1. Setup windows 2008 instances on two Regions, one in US and one in foreign country of your choice.

• Sign in to the AWS Management Console. (See instructions in logistics steps above.)
• The following steps will set up a windows 2008 server on US East (virgnia) region. You will repeat these for the foreign region.
• 
• 
• 
• 
• 
• make sure you save the downloaded private key for the key pair to a safe folder. AWS will not keep the private key in their system. You need this private key to retrieve windows root password or access linux server directly.
  
• Choose "default" security group which allows all ports. You can specify a security group that only allow rdp and http (80) access/
  
• 
• Click Close.
• Click the Instances menu in the Navigation panel to observe the status the new instance.
• 
• 
• 
• Use the domain name in remote desktop connection. In this case, it is "ec2-174-129-121-250.compute-1.amazonaws.com".
• 
• 
• Turn on IIS web service.
• Select Server Manager in Administrative Tools
  
• 
• Click next
• 
• Click next.
• 
• Click install
• Take about 15-20 minutes to complete the installation.
• 
• Click close
• 
• 
• After adding cs.uccs.edu to the trusted site list, you need to click the url again to download ssh.
• Use SSH Secure File Transfer to copy bmsite (benchmark site) from gandalf /home/cs526/public_html/cs526/bmsite to your windows 2008 server.
  • 
  • gandalf.uccs.edu can be accessed from instance (outside of uccs) while walrus.uccs.edu cannot be access from your instantce (it is behind uccs firewall)
  • Some of you think you can establish uccsvpn on your instance then access walrus like you access walrus at home. But it is not that easy. You will find the moment uccsvpn is established on your instance, you lose the RDP connection to your instance!! It is related to your instance now listen to all traffic on that new connection not the original connection. Bonus exercise: figure out how to configure instance to use a vpn connection.
• Now repeat this to setup a window 2008 server on a foreign region.

Benchmark the performance of two IIS servers you just set up.

• Use the same ab benchmark program on gandalf to evalute the performance of the IIS on windows 2008 server at N. Virginia EC2 site. Use "ab -c 20 -n 5000 http://ec2-174-129-121-250.compute-1.amazonaws.com/bmsite/ > virginiaR1.txt " where ec2-174-129-121-250.compute-1.amazonaws.com should be replaced with your instance's public DNS name.
• Use the same ab benchmark program on gandalf to evalute the performance of the IIS on windows 2008 server at the foreign site.
• When you are done with your session, make sure you stop both instances.
• Q1.1. Compare the performance of the two IIS servers. Discuss the reason for such results.
• Q1.2. There are four Availability Zones in N. Virginia site. If you have four servers, would you spread them in each of the four zones or set up them all in the same zone? What are the design trade-off?
• Q1.3. If have customers in the Saudi Arabia, which region of the Amazon data center will you choose to setup the web server? What information will you collect for making such a decison? How you obtain it?

Part 2. Use Amazon EC2 API Tools to control instances.

• See http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/
• ec2-api-tools: The command-line tools serve as the client interface to the Amazon EC2 web service.
  • Use these tools to register and launch instances, manipulate security groups, and more. http://www.rpmfind.net/linux/rpm2html/search.php?query=ec2-api-tools
• If you choose to work from home pc with Windows. You need to install ec2 command line tools for windows. See instruction from http://www.powercram.com/2009/12/installing-ec2-command-line-tools-on.html
• On gandalf, I have installed EC2 API command line tools using rpmfind with RpmFusion Non-Free repository. I also set the system environment variables EC2_HOME and EC2_PRIVATE_KEY and EC2_CERT. Therefore you do not have to set (export) those env variables.
• Login to gandalf or walrus.
• type "ec2-describe-regions" as a test. It will query aws web services with your cert and private key. The results will return in 5-6 seconds.

  [chow@gandalf ~]$ ec2-describe-regions 
  REGION  eu-west-1       ec2.eu-west-1.amazonaws.com
  REGION  sa-east-1       ec2.sa-east-1.amazonaws.com
  REGION  us-east-1       ec2.us-east-1.amazonaws.com
  REGION  ap-northeast-1  ec2.ap-northeast-1.amazonaws.com
  REGION  us-west-2       ec2.us-west-2.amazonaws.com
  REGION  us-west-1       ec2.us-west-1.amazonaws.com
  REGION  ap-southeast-1  ec2.ap-southeast-1.amazonaws.com
  

• Authorize access to instance. You can use the following command to grant access from your

  ec2-authorize default -p 22 -s your-local-system's-public-ip-address/32

  I have added ssh, http, and rdp service to default security group through management console.
• From the instance menu display results, we know that we have an instances with name=chowWin2008 and instance ID=i-7c842f13, AMI ID=ami-c3e40daa. To find out more details about this instance, we can use the following command:

  [chow@gandalf ~]$ ec2-describe-instances i-7c842f13
  RESERVATION     r-7629731b      664170878440    default
  INSTANCE        i-7c842f13      ami-c3e40daa    ec2-174-129-121-250.compute-1.amazonaws.com     ip-10-110-75-241.ec2.internal   running chow2key        0      m1.small 2011-03-31T07:57:42+0000        us-east-1b                      windowsmonitoring-disabled      174.129.121.250 10.110.75.241                   ebs    hvm
  BLOCKDEVICE     /dev/sda1       vol-0b959363    2011-03-31T07:58:03.000Z
      
      

• To stop an instance, (note that stop is not terminate)

  [chow@gandalf ~]$ ec2-stop-instances i-7c842f13
  INSTANCE        i-7c842f13      running stopping
      

• Use management console to verify if the status is "stopped".
  
  
• To start an instance,

  [chow@gandalf ~]$ ec2-start-instances i-7c842f13
  INSTANCE        i-7c842f13      stopped pending

• Verify from the managemnet console the instance is running. Use refresh button or reload button to see the changes of the status. Capture the instance entry as an image for the deliverable.
• Stop it again. Capture the instance entry again.
• Save both files in your cs526 web site and reference them in your hw5.html web page.
• Note that aws has 8 regions.  Without specifying the specific region, the ec2 command will look for default region setting and send the command to us-east-1 (N. Virginia).
  That is the reason why your command works when deals with instance in N. Virginia.
  For other region, you need to look for the command option which specifies the specific region.  In the case, we are working on instances at Singapore region we need to add --region ap-southeast-1 See attached example.
  See the related document http://docs.amazonwebservices.com/AWSEC2/latest/CommandLineReference/  in particular
  http://docs.amazonwebservices.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-DescribeInstances.html
  You can look the left side for specific command, their syntax and options.

  [chow@walrus ~]$ ec2-describe-instances --region ap-southeas-1 i-e16dc9b4
  Unknown host: 'https://ec2.ap-southeas-1.amazonaws.com'  (the option is used to form https:// request.)
  
  [chow@walrus ~]$ ec2-describe-instances --region ap-southeast-1 i-e16dc9b4
  RESERVATION     r-df429e8a      664170878440    cs526WinSG
  INSTANCE        i-e16dc9b4      ami-edec93bf                    stopped chowKey2011_5   0               m1.small        2011-05-08T00:54:17+0000        ap-southeast-1a                 windows monitoring-disabled                            ebs                                      hvm     xen             sg-50084202    default
  BLOCKDEVICE     /dev/sda1       vol-0d30e960    2011-05-14T19:03:47.000Z       true
  TAG     instance        i-e16dc9b4      Name    cs526_hw4_win2008
  [chow@walrus ~]$ ec2-describe-instances --region ap-southeast-1 i-e16dc9b4
  RESERVATION     r-df429e8a      664170878440    cs526WinSG
  INSTANCE        i-e16dc9b4      ami-edec93bf                    stopped chowKey2011_5   0               m1.small        2011-05-08T00:54:17+0000        ap-southeast-1a                 windows monitoring-disabled                            ebs                                      hvm     xen             sg-50084202    default
  BLOCKDEVICE     /dev/sda1       vol-0d30e960    2011-05-14T19:03:47.000Z       true
  TAG     instance        i-e16dc9b4      Name    cs526_hw4_win2008
      
      
      

Running Linux instance from command line and schedule its uptime:

• Now let us try to "run" an Linux instance from an AMI image.
• Select the EC2 dashboard menuitem on the left navigation panel and then click the "Launch Instance" button.
• The "create new instance" dialog window appear. Choose the "classic wizard" and hit conintue.
• we observe there are few instances with different LInux distribution. One of them with title Amazon Liinux AMI 2012.03. Let us chooa new Linux instance provided by Amazon.
  
• The "Request Instance Wizard" window appears. Choose the defaults and hit continue.
• In "instance detail" step, choose the default and hit continue.
• You can specify tags (key-value pairs) to associate (identify) the instance. Make sure you enter the value <yourlogin>AMLinux for the entry with key="Name". This will be used as instance name in the list.
• 
• 
• Save this keypair.pem file and upload to your aws directory on gandalf.
• Configuration Firewall: Just choose the default firewall setting. Click continue.
• Review the settings of your choice. Then hit "Launch".
• 
• Find out the instance ID and public DNS name for future access.
• 
• Run this instance with the following command from your aws directory.

  [cs526@gandalf aws]$ ls
  cert-4NR4SZJMIQM7N4VKKVDOFIDMY6L5QLUP.pem
  chow2key.pem
  pk-4NR4SZJMIQM7N4VKKVDOFIDMY6L5QLUP.pem
  [cs526@gandalf aws]$ ec2-run-instances ami-8c1fece5 -k chow2key.pem
  Client.InvalidKeyPair.NotFound: The key pair 'chow2key.pem' does not exist
     
     [cs526@gandalf aws]$ vi chow2key.pem
  -----BEGIN RSA PRIVATE KEY-----^M
  MIIEpAIBAAKCAQEAu5niiAnv1iOAN3Blcs5FR0ws/xVz6nthKIAIJA7JBqLqlEk3QzlQBhiI3Igg^M
  WszXy8efxFlfSXb2CiHH63PgrOn9UPdwAWZCNu/qKUAvYLWFDW13+6QBSWge+d8X+HNd/tAAFDVF^M
  ScMDrVxmIQiUKerefIBWUAO5PTWKXReRRg8WdBymaoVHf/M2Mt7tUofiXsOI4jQZFR09Zn+7lnZN^M
  yxLl7kQQV+kc+zmRyv1mvsEBoUA2LsqqBglrhEXKBS42MSaT3eVaRbEjIGbDdEWl6XdLYBEWeA1t^M
  BbiMVg4sZUK5DhIKgoU+VQQCGEUFq8RnHNxEueJ2fJuF5U12ylq5uwIDAQABAoIBAQCNp58D9ezh^M
  lyDju7z+uX5aXczEOCC098DfYf3DrF5i4nWazsYpvocMc040p502suJnfy7Ghm2K/Ya/E87vnSoT^M
  pwOeNZifleV9sh7wUwTzU11KTFSC8M3/7d0l9C0Xb5WUKN9GYwsUrWqDZve/DclC9ZRWEloaRRWE^M
  gH8f2AHcEY4dUZqUWx9/F/qNZXVKimJwwFQAHem4TLbkssEv5X4Pv6BDvW+FIHdMivly3WSJgCSG^M
  YfFdMZC5v3Qcc9Ad1P+cy+XWt1GJKuNe6QfYUepD93+PNhU4q
  
      

• The chow2key.pem show ^M at the end. It turns out these additional characters was part of DOS end of line file format. When we use secure file transfer to copy file over to gandalf which is a unix file system, the Unix interpret the CR character as part of the content before end of line.
• The line terminator expected for each file format is:

  unix	LF only (each line ends with an LF character).
  dos	CRLF (each line ends with two characters, CR then LF).
  mac	CR only (each line ends with a CR character).

  CR is carriage return (return cursor to left margin), which is Ctrl-M or ^M or hex 0D.

  LF is linefeed (move cursor down), which is Ctrl-J or ^J or hex 0A. Sometimes, LF is written as NL (newline). See http://vim.wikia.com/wiki/File_format

• Suspect that file format may cause the ec2-run-instances to misinterpret the private key file content.
• Try File conversion using dos2unix:

  [cs526@gandalf aws]$ man dos2unix
  dos2unix(1)                       2010-08-18                       dos2unix(1)
  
  NAME
         dos2unix - DOS/MAC to UNIX and vice versa text file format converter
  
  SYNOPSIS
             dos2unix [options] [-c CONVMODE] [-o FILE ...] [-n INFILE OUTFILE ...]
             unix2dos [options] [-c CONVMODE] [-o FILE ...] [-n INFILE OUTFILE ...]
  
  DESCRIPTION
         The Dos2unix package includes utilities "dos2unix" and "unix2dos" to
         convert plain text files in DOS or MAC format to UNIX format and vice
         versa.  Binary files and non-regular files, such as soft links, are
         automatically skipped, unless conversion is forced.
         
  [cs526@gandalf aws]$ dos2unix chow2key.pem
  dos2unix: converting file chow2key.pem to UNIX format ...
  [cs526@gandalf aws]$ vi chow2key.pem
  -----BEGIN RSA PRIVATE KEY-----
  MIIEpAIBAAKCAQEAu5niiAnv1iOAN3Blcs5FR0ws/xVz6nthKIAIJA7JBqLqlEk3QzlQBhiI3Igg
  WszXy8efxFlfSXb2CiHH63PgrOn9UPdwAWZCNu/qKUAvYLWFDW13+6QBSWge+d8X+HNd/tAAFDVF
  ScMDrVxmIQiUKerefIBWUAO5PTWKXReRRg8WdBymaoVHf/M2Mt7tUofiXsOI4jQZFR09Zn+7lnZN
  yxLl7kQQV+kc+zmRyv1mvsEBoUA2LsqqBglrhEXKBS42MSaT3eVaRbEjIGbDdEWl6XdLYBEWeA1t
  
  [cs526@gandalf aws]$ ec2-run-instances ami-8c1fece5 -k chow2key.pem
  Client.InvalidKeyPair.NotFound: The key pair 'chow2key.pem' does not exist
  

• Still have problem. Ah! It says key pair does not exist, not that file content is not correct!
• Further study on the ec2-run-instances option show -k is asking for the name of the key pair, not the file name of the key pair. http://docs.amazonwebservices.com/AWSEC2/latest/CommandLineReference/
  

  Syntax

  ec2-run-instances ami_id [-n instance_count] [-g group [-g group ...]] [-k keypair] [-d user_data |-f user_data_file] [--addressing addressing_type] [--instance-type instance_type] [--availability-zone zone] [--kernel kernel_id] [--ramdisk ramdisk_id] [--block-device-mapping block_device_mapping] [--monitor] [--disable-api-termination] [--instance-initiated-shutdown-behavior behavior] [--placement-group placement-group] [--tenancy tenancy] [--subnet subnet] [--private-ip-address ip_address] [--client-token token]

  Options

    
    -k, --key keypair
  	The name of the key pair.
     Type: String
     Default: None
     Example: -k websvr-keypair
  
  
  [cs526@gandalf aws]$ ec2-run-instances ami-8c1fece5 -k chow2key
  RESERVATION     r-f6e3c49b      664170878440    default
  INSTANCE        i-0269de6d      ami-8c1fece5                    pending chow2key0               m1.small        2011-04-01T03:13:42+0000        us-east-1c     aki-407d9529                     monitoring-disabled                            ebs                                      paravirtual
  [cs526@gandalf aws]$ ec2-run-instances ami-8c1fece5 -k cs526key
  RESERVATION     r-82e2c5ef      664170878440    default
  INSTANCE        i-7057e01f      ami-8c1fece5                    pending cs526key0               m1.small        2011-04-01T03:16:01+0000        us-east-1c     aki-407d9529                     monitoring-disabled                            ebs                                      paravirtual
  [cs526@gandalf aws]$ vi cs526key.pem
  -----BEGIN RSA PRIVATE KEY-----^M
  MIIEpAIBAAKCAQEAk/XXT84VrKz6raeRSXh/U+TRvrLuuLGzZaC1FPDl/SBSP//QHKTafv8BE2sz^M
  Ok8oWTGpx3sJx/qewEoQDBdVUofWrdHrp0whbYklCXHTUxZePl0VbZl43SknQzM2WQZTOUHisEOU^M
  D+8jcVrlsTydxzuXP199TOqTEI3H3WLCqVWffOlupyUfX9PQO+jOAxTkzTmrkziVt4VVjPmZBULh^M
  duneNjABpmc0kHCxSO2IIHZEEY86nGzIsEdnBvQwiO+vn4UfpVVkbh6+XnlfP6k8EFXnlF3aTdkH^M
  V4iS9s0iAQYW6Gx9WthKr0zTJC252iOIiwhy0y5jbHQ1YsgGcCyfLwIDAQABAoIBACNVq5TIb46s^M
  RLVh9iVonGfJlnmHO9FpxqD3jycwHIylwl68VdPxXqjDd1tO144OR2F9pcEY1g4nAPCE/24TyHGM^M
  NS2Pm0p5uOXywaCOyVvnVJxqGGrzItUWNeUrfsgl5Ywf+uYDTYG8p2czwPThuABTb9xXjpHm1zY1^M
  
         
         

• Verify the management console. It shows that two new instances does not have names but running.
  
• From the option description
  

  --availability-zone zone

  The Availability Zone in which to run the instance. Type: String Default: None Example: --availability-zone us-east-1a

• Use the following command to set the right availability zone to us-east-1b

  [cs526@gandalf aws]$ ec2-run-instances ami-8c1fece5 -k cs526key --availability-z
  one us-east-1b
  RESERVATION     r-4cd0f721      664170878440    default
  INSTANCE        i-142d9a7b      ami-8c1fece5                    pending cs526key0               m1.small        2011-04-01T04:34:38+0000        us-east-1b     aki-407d9529                     monitoring-disabled                            ebs                                      paravirtual
  
  

  
  

Access the Linux instance

• After you have created the Linux instance, find out the public DNS of the instance (ec2-107-22-13-108.compute-1.amazonaws.com)
  
  From gandalf/walrus in your aws directory with <login>KeyPair.pem (privateKey which you just uploaded for your Linux instances created during the instance creation; the public key is saved by amazon in ), run the following ssh -i command.

  [chow@walrus aws]$ ssh -i chow3KeyPair.pem ec2-user@ec2-107-22-13-108.compute-1.amazonaws.com
  The authenticity of host 'ec2-107-22-13-108.compute-1.amazonaws.com (107.22.13.108)' can't be established.
  RSA key fingerprint is 57:2a:cf:02:4b:72:d0:ca:c0:63:42:5c:e1:04:92:f0.
  Are you sure you want to continue connecting (yes/no)? yes
  Warning: Permanently added 'ec2-107-22-13-108.compute-1.amazonaws.com,107.22.13.108' (RSA) to the list of known hosts.
  
         __|  __|_  )
         _|  (     /   Amazon Linux AMI
        ___|\___|___|
  
  See /usr/share/doc/system-release/ for latest release notes.
  There are 2 security update(s) out of 19 total update(s) available
  Run "sudo yum update" to apply all updates.
  [ec2-user@domU-12-31-39-09-2D-8C ~]$ uname -a
  Linux domU-12-31-39-09-2D-8C 3.2.12-3.2.4.amzn1.x86_64 #1 SMP Thu Mar 22 08:00:08 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
  [ec2-user@domU-12-31-39-09-2D-8C ~]$ ifconfig eth0
  eth0      Link encap:Ethernet  HWaddr 12:31:39:09:2D:8C  
            inet addr:10.210.50.122  Bcast:10.210.51.255  Mask:255.255.254.0
  ...[ec2-user@domU-12-31-39-09-2D-8C ~]$ cd .ssh
  [ec2-user@domU-12-31-39-09-2D-8C .ssh]$ ls
  authorized_keys
  
  The authorized_keys file contains the public_key of the chow3KeyPair with the following content:
  
  ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDWybdfW1JYXzzxSc2NTpgb7jAyt4Uc0Hcd5qXwaUGFtEsNcs3PMwGE
  ppXZlcf4xv/YOf4VI8tcur6qm8RZ0cLg0oaL+RVu8GkcSKuVEGsrsNE12I8w0ARoahv/En+wj7vbWm2CW8QeyQvT5tnB
  SRUq9daXXXXXXXXXXXXLgd chow3KeyPair
  
        

• Save the above Linux login session as part of the deliverable for part 2 and reference it from your hw5.html.
• Q2. One of the most attractive features of EC2 is able to start the instance on demand. Assume that we know our students will only work on their server related exercises during evening hours (most of them have daytime work:-)), therefore we would like to run the Amazon Linux AMI instance only from 7-11pm.
  • Q2.1. Describe your design of a system with the script that will automatically schedule the instance to run every night from 4/19 to 4/30.
  • Q2.2. Assume you use ec2 command line api and have reserved an elasic IP address 50.17.160.64 for this instance. What will be the two start/stop ec2-api command looks like?
  • Q2.3. Elastic IP address remap and idle will be charged with the following expense. What will be the total estimated cost for using aws to provide services in the above period? Please include EBS volume charge, Elastic IP address charge, and on-demand instance charge.
    

    No cost for Elastic IP addresses while in use

    • $0.01 per non-attached Elastic IP address per complete hour
    • $0.00 per Elastic IP address remap – first 100 remaps / month
    • $0.10 per Elastic IP address remap – additional remap / month over 100
      
      
• Bonus exercise 1: Develop the above script and demonstrate it works.
• Bonus exercise 2: The API does not seem to provide ways to set Name of instances. The user_data option does not seem to set tag name-value pairs. See if you can find a good solution.
• Interesting info and resource:
  • Interesting ec2 management tool sample written in ruby. http://developer.idapted.com/2010/06/30/amazon-ec2-management-tool/
  • Stratus - Ruby IAM client library http://rubydoc.info/gems/stratus/1.0.1/frames

Part 3. Evaluate cost and performance trade off.

• Compared with the investment on the hardware/software of your own facility, the public cloud computing facilities have the following advantages and disadvantages:
  • Pros: On-demand provisioning of servers, with data center georgraphically distributed (closer to the customers/users), low cost and quick start up time.
  • Cons: One time charge unlike hardware or software investment can be reused for 3-4 years, less secure, hard to know if public cloud computing facilities really honor the service level agreement.
• For longer terms usage, it is also not clear whether the reserved-instance or on-demand usage will save more money. We need to consider the storage usage and elastic IP address mapping and inactive charge.
• Here is an example of cost calculation for an online degree offering using Amazon AWS and compared that with the private cloud facility.
• Assume you are setting up a site with one LInux server and one Windows 2008 server to provide computing and server facilities for a group of 60 students at Europe, students are allowed to access the system 6 hours during the evening for one month during their certificate program study. Recommend your design choice using Amazon AWS and calculate the cost (using excel spreadsheet). Make your own assumptions.