Using Chef to manage Amazon EC2 instances, Part 1

Gerhard Lazu • Tuesday 10 August 2010

So now that you have you Amazon AWS account, you must feel smug and think that you’re on the top of the cloud computing world. Not so quick tiger. Let’s say that you fire up an EC2 instance, what next? How are you going to configure it? Let’s tackle this step by step. I’m on OS X by the way, you should be too.

Keys, Certificates & authentication

Install brew if you don’t have it, then create an .ec2 folder:

$ brew install ec2-api-tools
$ mkdir ~/.ec2

Create a file in the newly created folder called e.g. gerhard_aws containing:


# Setup Amazon EC2 Command-Line Tools
export EC2_ACCESS_KEY="<your-access-key>"
export EC2_SECRET_KEY="<your-secret-key>"
export JAVA_HOME="/System/Library/Frameworks/JavaVM.framework/Home"
export EC2_PRIVATE_KEY="$(/bin/ls $HOME/.ec2/pk-*.pem)"
export EC2_CERT="$(/bin/ls $HOME/.ec2/cert-*.pem)"
export EC2_HOME="/usr/local/Cellar/ec2-api-tools/<your-version>/jars" # ec2-version w/o date

The Access Key, Secret Key and Certificate should be easy to find in your Amazon AWS account. You’ll find them in Account > Security Credentials. You should be given the Private Key when creating your first keypair, but I’m not 100% sure. You need to move this Private Key into the same .ec2 folder.

In your shell’s configuration file add the following and reload the shell environment:

if [[ -s $HOME/.ec2/gerhard_aws ]] then
  source $HOME/.ec2/gerhard_aws

After generating a new EC2 keypair, put everything starting with -----BEGIN RSA PRIVATE KEY----- into a new file called the same name as the keypair (ec2-keypair in my example). When done, let’s add this key to our agent so that we can ssh into our EC2 instances with no extra hassle:

$ ec2-add-keypair ec2-keypair
# this will output the new keypir contents
$ ssh-add ~/.ec2/ec2-keypair
$ ssh-add -l # make sure it was added!

As an extra tip, keep your .ec2 folder on Dropbox and just symlink it to your home folder:

$ ln -nfs ~/Dropbox/dotfiles/.ec2 ~/.ec2

Create Opscode account & install Chef locally

There is an excellent, up-to-date write-up on Getting Started with Chef, follow it!

Make sure that you install the following gems, they will save you a good ba… head scratch later:

$ gem install net-ssh-multi fog highline

At step 2, Create your Chef repository, I have created another folder on Dropbox and just symlinked it from there. I have also modified the cookbook path and added the AWS keys to .chef/knife.rb:

cookbook_path ["#{current_dir}/../site-cookbooks", "#{current_dir}/../cookbooks"]
# Amazon AWS
knife[:aws_access_key_id]     = ENV['EC2_ACCESS_KEY']
knife[:aws_secret_access_key] = ENV['EC2_SECRET_KEY']

We’ll create our first EC2 instance & configure it in the second part:
Using Chef to manage Amazon EC2 instances, Part 2