Encrypted Backup to Amazon S3

Posted:   |  More posts about System Administration

The following is a guide on how to do encrypted backups to Amazon S3.

Dt-S3-Backup Script

The dt-s3-backup.sh BASH script (http://blog.damontimm.com/bash-script-incremental-encrypted-backups-duplicity-amazon-s3/) combines the Duplicity backup utility, s3cmd, GPG and the Amazon S3 storage service to make a cheap, reliable and secure off-site backup storage solution.


You will require an Amazon S3 account (and a debit/credit card too).

  • duplicity (for incremental backups and Amazon S3 integration)
  • s3cmd
  • gpg (for encrypting the backup)

The new Amazon S3 interface is pretty good - but you may also use the S3Fox Firefox extension.


Amazon S3 and GPG keys

Once you have created your Amazon S3 account (http://aws.amazon.com/s3/) you will need to locate your Access keys (http://aws-portal.amazon.com/gp/aws/developer/account/index.html?action=access-key).

You have an Access Key ID, which is kind of like a Username and a Secret Access Key which is like a password. To see the Secret Access Key you need to click "show" and have JavaScript enabled.

Next you should make a GPG key to encrypt your backup. If you already have one you can use that but that means that you will need to have your private key on your server which isn't really desirable. You need to have the private key on the server because Duplicity needs to be able to both encrypt and decrypt the backup manifest so that it can do incremental backup.

To create a gpg key, run the following. It is best to do this on the server because that way you don't have to worry about importing or trusting.

# gpg --gen-key
Please select what kind of key you want:
   (1) DSA and Elgamal (default)
   (2) DSA (sign only)
   (5) RSA (sign only)
Your selection? 1
DSA keypair will have 1024 bits.
ELG keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 4096
Requested keysize is 4096 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0)
Key does not expire at all
Is this correct? (y/N) y

    ...The rest is up to you...

Make sure in the comment that you note that it is for your backups, so that you do not get it confused with your personal key.

After the key is finished generating (4096 bits worth of entropy takes a while!) then you need to export the private key and keep it in a safe place.

# gpg --list-keys
pub   1024D/6BBDC0C6 2010-06-01
uid                  Daniel Devine (just doing this as a test) <devine@ddevnet.net>
sub   4096g/424CF9A7 2010-06-01

# gpg --output DanielDevine_Backup_public.gpg --armor --export 6BBDC0C6
# gpg --output DanielDevine_Backup_private.gpg --armor --export-secret-key 6BBDC0C6

The number you need after the --export option will be different for you, so substitute your own by observing the output of "gpg --list-keys"

Now that you have exported your keys into a format you can transfer you should run an MD5 sum on it before you transfer it off the server.

# md5sum DanielDevine_Backup_private.gpg
4343afa0bccb0878e123209766687614  DanielDevine_Backup_private.gpg

Then you need to transfer it off your server. I prefer to use SFTP for this because it is easiest and most secure. If you have a SSH server running on the remote machine then you have SFTP!

# sftp [YourUsername]@[your IP address]
> PUT DanielDevine_Backup_private.gpg

> exit

On the computer that you transferred the file to do a md5sum on the file and check if the output matches what it did on your server. If it doesn't then transfer the file again because the first copy you did is corrupt and you can't use it to decrypt your backup!

Then delete the exported private key from the server. (use "rm DanielDevine_Backup_private.gpg")

Setting Up the Script

You can find the script at: http://github.com/thornomad/dt-s3-backup

or you download it and keep it up to date by using git.

git clone git://github.com/thornomad/dt-s3-backup.git

Once you have it you can copy/move the script to where you prefer to keep your backup/cron scripts. I just put it in a directory in root's home called "backupscripts". If you used git to obtain the script then it is best to copy rather than move. When you see than an update of the script is available you can use "git fetch" to get a new copy which you can merge your configuration into.

Before configuring the script use your distribution's package manager to get the dependencies for the script. One CentOS 5.5 with RPMFusion repositories you can run:

# yum install duplicity
# yum install s3cmd
# yum install gpg

The package names on Ubuntu/Debian are apparently the same. They should also be the same on Fedora.

On CentOS 5.5 the version of Boto is not new enough and will result in an error most likely to do with not being able to handle newer Amazon buckets. Download the newest version of Boto and install it.

# wget http://boto.googlecode.com/files/boto-2.0b3.tar.gz
# tar xzvf boto-2.0b3.tar.gz
# cd boto
# python setup.py install

Now to finally configure the backup script!

vim /root/backupscripts/dt-s3-cmd.sh

Read the comments and fill in the variables. If you want more information on the variables look at the Duplicity man page ("man duplicity") for a better description. If it asks you to go through Duplicity setup then you must do that. Just follow the prompts.

Next you should add an entry to crontab. The following is a good little snippet you can add to the top of the file to guide you through scheduling tasks.

# *     *     *   *    *        command to be executed
# -     -     -   -    -
# |     |     |   |    |
# |     |     |   |    +----- day of week (0 - 6) (Sunday=0)
# |     |     |   +------- month (1 - 12)
# |     |     +--------- day of        month (1 - 31)
# |     +----------- hour (0 - 23)
# +------------- min (0 - 59
# Remember to fill with 0's where appropriate. * means all so if you have * seconds its every second.

Else you can also use "@hourly" "@daily", "@weekly" etc. instead of filling out an exact time. I suggest backing up at least daily - it really doesn't take up much backup space if you configure dt-s3-backup.sh to do incremental backups.


If you get an error such as "Time too skewed" it is because your time is too different (mine was 2 hours in the future) from Amazon's as it can only be 9000ms different. This is a security thing on Amazon's part.

To get your system time right, install ntp.

# yum install ntp

To see the time difference between your system time and your BIOS clock:

# hwclock

Then to set up NTP do.

# ntpdate pool.ntp.org
# /etc/init.d/ntpd start

To update your BIOS clock.

# hwclock --systohc

Then you should make sure NTP comes up on startup.

# chkconfig nptd on

If you imported an existing key and you get an error to the effect of "There is no assurance this key belongs to the named user" then you probably need to set the trust of the key by doing the following.

# gpg --edit-key [Key ID - eg. DS8F978D]
> trust
> 5 [or whatever the level for ultimate trust is]
> save [not sure if this is needed]
> quit)
Comments powered by Disqus
Contents © 2013 Daniel Devine - Nikola Powered - Flattr Me! Flattr this Source