You are here: Home \ Portal Home \ Knowledgebase

Knowledgebase

Language:

Please Login or Register

Knowledgebase Article

Subject: How to set up Duplicity (secure incremental backup)?

Duplicity is a tool to create GPG-encrypted (this way you can store
your backups at remote servers without having to worry about who has
access to your data) incremental backups to remote servers. Its a quite
handy and secure method.

Installation

The steps to install duplicity is as follows

wget http://code.launchpad.net/duplicity/trunk/0.6.02/+download/duplicity-0.6.02.tar.gz

tar -xvf duplicity-0.6.02.tar.gz

cd duplicity-0.6.02.tar.gz

python setup.py install

Now we got duplicity installed.

Create a GPG key

In order to be able to encrypt your backups, you have to create a GPG
key. Open a second shell and run the following command (this generates
some βrandomnessβ on your system, which will be useful to create a
secure key). Kill the command with CTRL+C when you are done with key
generation.

while /bin/true; do cat /var/log/messages > ~/temp.txt; sleep 1; done;

On your other shell, create your GPG key. Be sure to use a secure
passphrase and to copy/write down the key ID which is displayed at the
end of the generation process (weβll need it for ftplicity). Also, make
sure to backup the key to a secure location outside your server. As all
your backups will be encrypted, they will be worthless if your server
crashes and you lose the key.

gpg βgen-key

Default options should be fine. This will create your key in ~/.gnupg/.
Once its done you can verify the existence of your key using the command

gpg βlist-keys

The next step is to prepare an off-site location to receive the backup files.

The software supports different protocols like FTP,RSYNC,SCpP.

I am restricting myself with SCP here

Simple unEncrypted Backup over SCP

Setup ssh keys on the backup server allowing root to seamlessly login to the backup server.

duplicity /home/me scp://uname@other.host/usr/backup

- If the above command is run repeatedly, the first session will be a full backup, and subsequent ones will be incremental.

The full option can be used to force a full backup. The next command also excludes the /tmp directory.

duplicity full βexclude /tmp /home/me scp://uname@other.host/usr/backup

- Basic restore commandβrestore the /home/me directory backed up with scp above to directory restored_dir:

duplicity scp://uid@other.host//usr/backup restored_dir

To enable verbose mode use the option -v

Specify verbosity level (0 is total silent, 4 is the default, and 9 is noisiest)

The command would look like

duplicity -v5 /home/me scp://uid@other.host/some_dir

Encrypted Backup over SCP

Here we use the GPG key generated earlier

The format would be look like this

duplicity

--encrypt-key=${GPG_KEY} \

--sign-key=${GPG_KEY} \

--include=/boot \

--include=/etc \

--include=/home \

--include=/root \

--include=/var/lib/mysql \

--exclude=/** \

${SOURCE} ${DEST}

Needless to say the include and exclude options are for specifying the backup criteria.

duplicity βencrypt-key=βFFF7730Bβ βsign-key=βFFF7730Bβ -v5 /home/me scp://uid@other.host/some_dir

you will be asked for a GnuPG passphrase. You can type in any password
you like; this has to be done everytime you run duplicity. The backup
will be encrypted with the help of GnuPG. Permissions and ownerships
will be preserved in the backup.

To avoid this issue , you can simply set the passphrase as environment variable using the command

export PASSPHRASE=gpgpassphrase

Backup Format & Explanation

Once it is executed , you can see the backup in the server and it would look like the following way

duplicity-full-signatures.2005-11-27T01:00:01-05:00.sigtar.gpg

duplicity-full.2005-11-27T01:00:01-05:00.manifest.gpg

duplicity-full.2005-11-27T01:00:01-05:00.vol1.difftar.gpg

duplicity-full.2005-11-27T01:00:01-05:00.vol2.difftar.gpg

The signatures file contains, signatures of each file that is backed up
so that Duplicity can figure out which part of a file has changed. With
that information it can upload only the missing part to complete a new
backup set.

The manifest file contains a listing of all the files in the backup set
and a SHA1 hash of each file, probably so Duplicity can tell very
quickly whether a file has been changed or not since the last backup.

The volume files (vol1 and vol2) contain the actual file data. It
appears that Duplicity volumes are at most 5MB. Thatβs helpful during
restores so the entire backup set doesβt not need to be downloaded to
retrieve a single file. Duplicity will only download the volume
containing that file.

Common Options:

Depending on the parameters and order of the parameters in the
duplicity command, different functions can be performed. For example,
an archive can be verified to see if a complete backup was made and
what files, if any, have changed since the last backup.

duplicity verify [options] source_url target_directory

duplicity verify -v4 scp://user@bakuphost/etc /etc

Listfiles

Itβs sometimes handy to check which files are in the latest backup set.

duplicity list-current-files [options] target_url

The command would look like

duplicity list-current-files βarchive-dir /root/test/ scp://user@backupserver/some_dir

Restore

The main purpose of backup is to restore data which has been lost. The
following is the common format for restoring the data from the latest
backup

duplicity scp://uid@other.host/some_dir /home/me

Duplicity enters restore mode because the URL comes before the local
directory. If we wanted to restore just the file βMail/articleβ in
/home/me as it was three days ago into /home/me/restored_file:

duplicity -t 3D βfile-to-restore Mail/article scp://uid@other.host/some_dir /home/me/restored_file

The following command compares the files we backed up, so see what has changed since then:

duplicity verify scp://uid@other.host/some_dir /home/me

The following command can be used to retrieve a single file from backup

duplicity βencrypt-key ββ βsign-key ββ βfile-to-restore home/sburke/file.txt scp://user@server.com/backup/ /var/tmp/file.txt

1. The path to the file that is to be restored is relative to the
directory on which the backup set is based. So in the command above,
home/sburke/file.txt plus the directory on which we based our backup
(/backup) equals /backup/home/sburke/file.txt/. It would not work to
put /backup/home/sburke/file.txt as the source path because the backup
will not recognize /backup as a valid path. The last portion in the
above command is the location where the file will be restored.

To delete old backups, we can use the following command

duplicity --full --remove-older-than 1Y /media/data/backup scp://uid@server/personal

To automate the tasks, you can write a shell script

Back

Quick Navigation

Portal Home

Client Area

Announcements

Knowledgebase

Submit Ticket

Downloads

Order

Knowledgebase

Useful Information

Quick Navigation