Automatically Sync EC2 to S3

AWS Nomads #3: A quick win — a script to keep your Cloud9 / EC2 working directory in sync with your public S3 bucket.

They are smiling because their EC2 instances are in perfect harmony with their S3 buckets.

First of all, do not use this while working directly on a website that has traffic. You are going to save files, possibly accidentally, and they are going to be dropped into your public bucket instantly, and you are going to be making mistakes and breaking things.

If you don’t already have some form of staging system or development / production environment going for your live website, please consider that. (Note to self: future article.)

But while you are in the initial phase of rapid development or if you are syncing to a private staging environment that is not visible to the general public, this is a very pleasant and robust utility — and one that I have found many inferior or incomplete solutions for on StackOverflow and other blogs.

The gist of it (haha, yes the whole script is available in a gist, but also included below) is this:

1. We give the local directory the same name as our s3 bucket.
2. We run this script from the parent of our local directory with that bucket name.
3. The script uses inotifywait to watch for changes on the developer box. EC2 does not come with inotifywait installed, so you need to do so yourself:
   sudo yum install inotify-tools
4. With the -m parameter inotifyway watches the directory you give it, and writes changes to standard out.
5. We pipe standard out to a while loop that reads the line, parses it into operation and filename.
6. We take appropriate action on the file.

Without further ado:

A few additional notes:

• This only syncs to S3. If you make changes to S3 by other means, you will need to manually pull those changes down, preferably before running this script and accidentally blowing a change away.
• This is particularly true of images! If you upload images to S3 by other means, make sure to copy them locally first. (An exercise for the adventurous would be a two way sync, or an option to sync down the bucket on load, etc.) (Hmm. I like that, maybe I will update it?)
• If things get out of kilter somehow, don’t be afraid to take matters into your own hands with the aws client. It’s very easy to use and well documented.

An early implementation of S3.