access:amazon_cloud_computing
Differences
This shows you the differences between two versions of the page.
| Next revision | Previous revision | ||
|
access:amazon_cloud_computing [2009/11/25 18:36] mmatthews created |
access:amazon_cloud_computing [2012/10/10 17:08] (current) |
||
|---|---|---|---|
| Line 16: | Line 16: | ||
| Amazon's EC2 service allows customers to dynamically create and deploy Linux or Window servers. You pay an hourly rate based on the performance of the server. A basic server with 1.7gb of memory and an approx. 1.5ghz equivalent processor runs $0.085/hour. Prices climb quickly as you require higher performance machines. You can select from a huge variety of server configurations. You can also modify the configuration, create a bootable image from it, store the image in S3 and directly boot from that. For example, I started with the most basic Fedora 8 configuration they had and installed Sun's Java 1.6 distribution (required by the Agtek Access Server). I created an Amazon AMI image directly from the machine and stored it to our S3 account. I can now use that image anytime I want to launch a new server. There is local storage associated with the running server but it is NOT persistent. To store persisted data your application must use either S3 or EBS. | Amazon's EC2 service allows customers to dynamically create and deploy Linux or Window servers. You pay an hourly rate based on the performance of the server. A basic server with 1.7gb of memory and an approx. 1.5ghz equivalent processor runs $0.085/hour. Prices climb quickly as you require higher performance machines. You can select from a huge variety of server configurations. You can also modify the configuration, create a bootable image from it, store the image in S3 and directly boot from that. For example, I started with the most basic Fedora 8 configuration they had and installed Sun's Java 1.6 distribution (required by the Agtek Access Server). I created an Amazon AMI image directly from the machine and stored it to our S3 account. I can now use that image anytime I want to launch a new server. There is local storage associated with the running server but it is NOT persistent. To store persisted data your application must use either S3 or EBS. | ||
| - | ==== Amazon RDS - Relation Database Service ==== | + | ==== Amazon RDS - Relational Database Service ==== |
| Amazon's RDS service is similar to EC2 except that it is completely focused on providing MySQL 5.1 compliant servers. You pay an hourly rate which is slightly higher but similar to the EC2 rate to run a fully configured and managed MySQL 5.1 server. Additionally you pay a monthly fee for the database storage associated with the server. Included in the cost is a nightly backup with single day retention. Multiple day retention is available but may cost extra (depends on storage requirements). For my tests I used a basic machine with 5gb of persistent storage. | Amazon's RDS service is similar to EC2 except that it is completely focused on providing MySQL 5.1 compliant servers. You pay an hourly rate which is slightly higher but similar to the EC2 rate to run a fully configured and managed MySQL 5.1 server. Additionally you pay a monthly fee for the database storage associated with the server. Included in the cost is a nightly backup with single day retention. Multiple day retention is available but may cost extra (depends on storage requirements). For my tests I used a basic machine with 5gb of persistent storage. | ||
| - | ==== Amazon EBS - Elastic Block Storage ==== | + | ==== Amazon EBS - Elastic Block Store ==== |
| - | Amazon's EBS service allows customers to buy the equivalent of a raided disk system and attach it to a running EC2 server. Disk sizes range from 5gb to 1tb. They claim that reliability is 10 times that of a standard raid disk system but the details are a bit sketchy. If additional reliability is required the EBS can be backed up to your S3 account. The advantage of EBS over S3 is that EBS can be formatted to act just like a local hard disk. I did not test this product as I didn't need persistent storage for the tests. | + | Amazon's EBS service allows customers to buy the equivalent of a raided disk system and attach it to a running EC2 server. Disk sizes range from 5gb to 1tb. They claim that reliability is 10 times that of a standard raid disk system but the details are a bit sketchy. If additional reliability is required the EBS can be backed up to an S3 account. The advantage of EBS over S3 is that EBS can be formatted to act just like a local hard disk. I did not test this product as I did not need persistent storage for the tests. |
| ==== Amazon S3 - Simple Storage Service ==== | ==== Amazon S3 - Simple Storage Service ==== | ||
| - | Amazon's S3 service is the backbone of the whole system. S3 provides long term, fully reliable data storage. All the other products fall back on S3 for reliable storage. For example, if you create your own AMI image to boot an EC2 server Amazon requires that the image is stored in S3. All the RDS and EBS data backups are done to S3. We are currently using S3 for backing up various Agtek machines. | + | Amazon's S3 service is the backbone of the whole system. S3 provides long term, fully reliable data storage. All the other products fall back on S3 for reliable storage. For example, if you create your own AMI image to boot an EC2 server Amazon requires that the image is stored in S3. All the RDS and EBS data backups are done to S3. We are currently using S3 for backing up various Agtek machines (through Jungle Disk). |
| + | ===== Investigation ===== | ||
| + | |||
| + | Our goal is to eventually (hopefully in the next month) move all Agtek server products out of the lab here in Livermore and into some hosting/cloud solution. Keeping that goal in mind I set out to try the following with the Amazon products: | ||
| + | |||
| + | - Start an Amazon EC2 instance running an acceptable version of Linux | ||
| + | - Install the Agtek Access Server and database on the instance | ||
| + | - Run, configure and test the server with our standard 20 GPS load test | ||
| + | - Start an Amazon RDS instance to host the Access Server database | ||
| + | - Start an Amazon EC2 instance running just the Access Server, no database | ||
| + | - Run, configure and test the server with our standard 20 GPS load test | ||
| + | - Create a custom EC2 image containing everything we need for our server. | ||
| + | - Start an Amazon EC2 instance running our custom image. | ||
| + | - Sanity check all the various Amazon charges | ||
| + | |||
| + | ==== Running An EC2 Instance ==== | ||
| + | |||
| + | The hardest part of running an EC2 instance is finding the configuration you want. There are tons of configurations and they aren't well indexed. I ended up resorting to Google searches and never did find the exact one I wanted. Finally I gave up and just launched a basic Fedora 8 configuration that contained a full LAMP stack. There is a lot of security issues that have to be addressed the first time you launch an EC2 instance, but once those are understood and handled they are taken care for all future instances. Within about 2 hours I was able to figure out the whole thing, launch an instance, install our server and run the basic load test. In this test the Access Server and MySQL database were both on the EC2 instance. With both running on the smallest/cheapest EC2 available CPU utilization did not go over 25% while running the load test and memory usage did not go over 50%. | ||
| + | |||
| + | ==== Running EC2 and RDS Instances Together ==== | ||
| + | |||
| + | A better solution for our server would be to offload the database to a dedicated and managed server and just use the EC2 instance to run the Access Server and store the track data. There is a bit of a chicken and egg issue that occurs at this point. In order to minimize network bandwidth charges you need to have both the EC2 and RDS instances running in the same Amazon Availability Zone. This is much easier to do with the RDS than with the EC2. However, you need to know the address of the RDS before you can start the Access Server. I ended up going through the following steps: | ||
| + | |||
| + | - Launch an EC2 instance | ||
| + | - Install our server software | ||
| + | - Launch an RDS instance in the same availability zone as the EC2 | ||
| + | - Get the RDS ip address | ||
| + | - Configure the Access Server to use the RDS for database storage | ||
| + | - Start the Access Server | ||
| + | |||
| + | Once I got all this configured and set up I ran the standard 20 GPS load test. Peak CPU usage on the EC2 was 15% and memory usage never exceeded 33%. Peak CPU usage on the RDS was 7%, I could not find the stats for memory usage. In both cases I was using the smallest, cheapest servers available. | ||
| + | |||
| + | ==== Creating A Custom Image ==== | ||
| + | |||
| + | Finally, I set out to create a custom server image containing a basic Fedora 8 configuration with the Sun JDK that the Access Server required. This took more time than all the other testing combined! The fault lies in a combination of out of date documentation on the Amazon website and a bug in their web based management console. I did finally get it working and was able to launch a new EC2 instance running the custom server image I built. | ||
| + | |||
| + | It did expose a non-trivial issue. Unless you explicitly pay for Amazon tech support (starts at $100/month) the only support you get is the online documentation and the user forums. There is NO e-mail address you can send a question to. Additionally, many of the features, such as the web based management console, are listed as BETA.....and they should be. In fact, the entire RDS service is listed as BETA as Amazon only began offering it in October of this year (yes, last month). | ||
| + | |||
| + | ==== Costs ==== | ||
| + | |||
| + | All my testing over the last two days including starting and stopping EC2 and RDS instances, uploading large amounts of software multiple times (Access Server code and Java distributions) and running the load tests totaled out at less than $2.00. That is the nice thing about this system, if you aren't using resources you aren't getting charged for them. You also have to be careful. When I first ran the EC2/RDS combo the instances were in different "Amazon Availability Zones" and we were getting charged for all transactions between the server and the database. If the two instances are in the same zone there would have been no charge (those charges only totaled $0.02). | ||
| + | |||
| + | ===== Final Analysis ===== | ||
| + | |||
| + | In the end I was able to get our Access Server running on a distributed system utilizing 1 EC2 and 1 RDS. Performance seemed comparable to what we get out of our lab systems, maybe even better as Amazon has a bigger network pipe than we do. Given the CPU and memory loads shown by our simple load tests I think we could use the smallest servers for the loads we'll be experiencing in the near future. | ||
| + | |||
| + | A basic system with 1 EC2 server, 1 RDS server and plenty of disk space for both would cost well under $200/month. As we sell the service we can easily upgrade to bigger, faster, more expensive servers. | ||
| + | |||
| + | The only downside to the Amazon solution is the lack of great management tools. There is a beta quality web based management console for the EC2 service, but NONE for the RDS service. RDS services have to be created and monitored using command line utilities. There may be some third party solutions here but I have not looked into that yet. This problem might be minimized if there was better tech support. But at a minimum of $100/month for the basic tech support package, that does not seem like a great answer. | ||
| + | |||
| + | To sum up: | ||
| + | |||
| + | ==== The Good ==== | ||
| + | - Easy to get up and running | ||
| + | - No hardware on our end | ||
| + | - Not using our network bandwidth | ||
| + | - Database backup, maintenance all handled by Amazon | ||
| + | - Costs scale according to our use | ||
| + | - No upfront costs makes it cheap and easy to build and test a system | ||
| + | - Easy to scale up systems | ||
| + | |||
| + | ==== The Bad ==== | ||
| + | - GUI based management in beta for EC2 | ||
| + | - No GUI based management for RDS | ||
| + | - No free tech support | ||
| + | |||
| + | ==== The Ugly ==== | ||
| + | - Need to have some basic linux system administration skills to manage everything | ||
| + | - Too many things have to be done by hand, using command line utilities | ||
| + | - Only free support through user forums and online documentation | ||
| + | - Some of the online documentation is out of date and incorrect | ||
access/amazon_cloud_computing.1259174196.txt.gz · Last modified: 2012/10/10 17:08 (external edit)
Page Tools
