I was recently introduced to Amazon's new Ec2 services.
The idea of cloud computing really intrigued me after I heard about it
so I decided to take the dive. There is a bit of a learning curve with
getting started but once you get started you realize the unlimited
potential that cloud computing offers. Ec2 offers the ability to deploy
pre-configured (linux based) images (called AMI's). The AMI's can be
created from scratch or based on prebuit versions that Amazon or other
users have exposed. You can quickly deploy to several different types
of machines depending on your requirements. The base system has a
1.7Ghz Xeon CPU, 1.75GB of RAM, 160GB of local disk, and 250Mb/s of
network bandwidth. Currently this will cost you $.10 per computing hour
plus bandwidth costs. You are only charged for the time that the
virtual machine is running and you can start and stop multiple
instances at your will to scale as you need to. There are also beefier
64-bit machines available at a higher cost. On limitation (depending on
how you look at it) is that persistent storage is not offered on the
instances. After you start it up if at any time it crashes you lose
everything on the instance. There are ways to overcome this as I will
explain later but it makes things a bit more challenging. I found that
the simplest way to get started is to find a public AMI that meets you
needs, make the modifications to the instance then save it as your own
instance into Amazon S3. S3 is another service that Amazon offers for storage, S3 and Ec2 work hand-in-hand with one another.
To get started you will need an account with Amazon Web Services at http://aws.amazon.com.
You will need to sign up with both Ec2 and S3. It does not cost
anything up front but you will need a credit card for them to draw
funds from once you start using the service. One thing that took me a
little while to get use to was the extensive use of certificates for
authentication. Beyond signing in to your AWS account nearly everything
else with the Ec2 service uses certificates or private keys. You use
them to start your instances, as well as gain remote root access to an
instance that you have started. It really makes things more secure. So
lets get started....btw I recently switched from PC to Mac so all of
the instructions will be for the Mac but they translate easily to the
PC if you are familiar with java.
- Log into your AWS account, I am assuming you signed up with Ec2 and S3 already.
- After you are signed click on the "You Web Services Account" button and you will find the "AWS Access Identifiers" link.
- Select X.509 certificates link.
you click on the "create new" link you will be asked to confirm, click
yes and the two files will be generated. You will find the two
following files. These are the certificates I mentioned above that are
used to authenticate you when any commands are issued to Ec2. There
will be an additional cert that we create later to launch your
- X.509 certificate named cert-xxxxxxx.pem
- RSA private key named pk-xxxxxxx.pem
- Next you will need to download the Amazon Ec2 command line tools.
- Now it is time to setup your machine to use the Ec2 tools.
- Open the terminal and go to your Mac home directory and create a new folder named ~/.ec2
- Copy the cert-xxxxxxx.pem and pk-xxxxxxx.pem into your ~/.ec2 directory from above.
the tools into the ~./ec2 directory and move out the bin and lib
directories to this directory as well. It should look like the
- cert-xxxxxxx.pem file
- pk-xxxxxxx.pem file
- The bin directory
- The lib directory
you will need to set a few environmental variables. To make things
easier you can place these changes in your ~/.bash_profile file. If
this file does not exist in your home directory you can create it then
add the following:
# Amazon Ec2 tools
export EC2_PRIVATE_KEY=`ls $EC2_HOME/pk-*.pem`
export EC2_CERT=`ls $EC2_HOME/cert-*.pem`
- After making the changes you will need to reload your ~/.bash by running the command
you are ready to start issuing commands to Ec2, list instances and
start them. The first step is finding the instance that is appropriate
for your needs. You can test with the amazon images that are available
and customize them to your needs. To list all of the Amazon instances
type the following command.
$ ec2-describe-images -o amazon
IMAGE ami-20b65349 ec2-public-images/fedora-core4-base.manifest.xml amazon available public
IMAGE ami-22b6534b ec2-public-images/fedora-core4-mysql.manifest.xml amazon available public
IMAGE ami-23b6534a ec2-public-images/fedora-core4-apache.manifest.xml amazon available public
IMAGE ami-25b6534c ec2-public-images/fedora-core4-apache-mysql.manifest.xmlamazon available public
IMAGE ami-26b6534f ec2-public-images/developer-image.manifest.xml amazon available public
IMAGE ami-2bb65342 ec2-public-images/getting-started.manifest.xml amazon available public
IMAGE ami-36ff1a5f ec2-public-images/fedora-core6-base-x86_64.manifest.xmlamazon available public
IMAGE ami-bd9d78d4 ec2-public-images/demo-paid-AMI.manifest.xml amazon available public A79EC0DB
of this bunch you should find at least one suitable to test with, we
will use the Fedora Core 4 machine with Apache from above. Before doing
this we need a keypair to start the instance. This keypair will be used
to gain root access to the instance through SSH after it is up and
- To generate the keypair use the following
command, this will create a RSA private key and output it to the
screen. You will copy this entire key from ------BEGIN RSA PRIVATE
KEY------ TO ------END PRIVATE RSA KEY------. Paste this into a new
file named ec2-keypair in your ~/.ec2 directory.
$ ec2-add-keypair ec2-keypair
step is something that I missed at first and it frustrated me until I
figured out what I was doing wrong. Before you can use this key to SSH
to a running instance the Ec2 tools require that you set permissions on
the file so that only your account has access to the file. You can do
that with the command.
$ chmod 600 ec2-keypair
- Now we can boot up an ec2 instance. We have chosen the ami-23b6534a instance from above. You will use the following command to start the instance.
$ ec2-run-instances ami-23b6534a -k ec2-keypair
will take a little while for your instance to start but while you are
waiting you can check on the status of the instance with the following
Once it is
up and running you will see "running" as the status. Take note of the
server addresses that this command provides since the provide the DNS
addresses you will need to access your instance with a web browser or
via SSH. They will be in the format of:
ec2-xx-xxx-xxx-xxx.compute-1.amazonaws.com - (Externally accessible DNS address)
domU-xx-xxx-xxx-xxx.compute-1.internal - (Internally accessible DNS address used from instance to instance)
server instances are locked down pretty tight and you will not have
external network access to any of the instances by default. You have
control over opening the ports though similar to controlling your own
firewall. The network access is not configured uniquely to each
instance but instead you control it by groups. You can launch several
instances in the same group and provide network access to that group.
When you start an instance like we did above it is started as part of
the "default" group. We now need to open up network access for web
traffic on port 80 and SSH on port 22 with the following commands:
ec2-authorize default -p 22
ec2-authorize default -p 80
- You can now access your instance by opening up your web browser and entering your address http://ec2-xx-xxx-xxx-xxx.compute-1.amazonaws.com
you are ready to access the command line of the instance. This is where
the private key that you created early comes in. You do not have a root
password, instead you use the private key to authenticate yourself. You
can access via SSH with the command:
ssh -i ec2-keypair firstname.lastname@example.org
Now you are up and running with your instance. You can change
whatever you want and add software to the Linux image. Just remember
that it does not persist if you shutdown. If you do a reboot it will
persist. After you have made all of the changes you want you can
repackage the instance as your own and store it into the Amazon S3
service (LINK TO THESE STEPS)
Challenges of working with Ec2
- You get a dynamic IP address each time you boot an image. There are solutions with DynDNS that are worth exploring.
is no persistent storage if an instance fails. There are ways to
overcome this limitation. So far I have worked with PeristantFS which
allows you to mount a bucket from S3 as a directory in your image.
are limited by space in the image to 10GB (I think I need to confirm)
if are going to store large files I suggest putting them somewhere in
the /mnt directory since that has a lot more space. Also if you save
the image anything in the /mnt folder is not saved as part of the
image. You can put log files and other content that you don't want
saved in this location
- Databases are a challenge with
limited options for persistence. Third parties are popping up offering
db hosting on the cloud so you don't have to manage it yourself. I will
explore these more in the future.
The future of scalable computing....
I really feel like cloud based solutions are the future for hosted
solutions. Once you work out some of the limitations you can build a
very scalable solution where you have automated scripts that launch new
instances as you have a need to scale. In turn you can shut them down
as the load decreases. There are overall architecture needs that have
to be addressed to utilize an infrastructure like this but it is all
doable with a bit of ingenuity. Add in the fact that a small business
does not have to invest an significant amount into hardware and
software to start running on this type of solution and it is a no
brainer. The questions of SLA's come up and I expect that to be an
issue for the short term but solvable in the future.
Getting started is easier with RightScale.com
I also used RightScale when I first got started with Ec2, they are a
third party that puts a front end onto the managing of ec2 instance. It
makes it a lot easier to get started and get your head around Ec2. All
you need is an AWS account with Ec2 and S3 and you can get started with
RightScale. You do not have to deal with all of the command line stuff
above and the Ec2 tools.
Good and consized information for newcomers,By the way is there a way to configure more than 1 public address to a single instance ?.
super ! thanks a lot, it clarified a lot of 'in the clouds' points for me....
step 20 I had to specify the .ec2 directory
ssh -i .ec2/ec2-keypair email@example.com