chickadee » ugarit-backend-s3

Outdated egg!

This is an egg for CHICKEN 4, the unsupported old release. You're almost certainly looking for the CHICKEN 5 version of this egg, if it exists.

If it does not exist, there may be equivalent functionality provided by another egg; have a look at the egg index. Otherwise, please consider porting this egg to the current version of CHICKEN.

ugarit-backend-s3

Description

This egg provides a Ugarit storage backend for Amazon S3 and other stores with an S3-compatible API.

Requirements

Usage

To make use of this egg, you'll first need an account for an object store with an S3-compatible service provider. This egg has been tested with PCextreme's Aurora Objects offering (which uses Ceph) and it should of course also work with the original S3 storage from Amazon. If you use another provider and run into problems, please let me know.

Next, you create two files. One is the Ugarit configuration for this backend (see the recommendations section below for more information about some choices):

(storage "sudo -u backup /usr/local/chickens/ugarit/bin/ugarit-backend-s3 /etc/ugarit/s3.conf")
;; Without sudo, it should just look something like this:
;;(storage "ugarit-backend-s3 /etc/ugarit/s3.conf")
(cache "/etc/ugarit/s3-cache.sqlite")

;; You can generate these hashes with the following command:
;; `head -c 64 /dev/urandom | base64 -w0'
(hash tiger "EsqAF7cTyrmo1cQjamjKo/nU5gENPaNtpcLJgn4OYbkLI+swCGl+Hddi5iDv7qijphvpJQsDI4WZOlvMouvVqQ==") ; requires the "tiger-hash" egg
(encryption aes (32 "uE0QVQJHZboV120HxcO/8DB06oN6qNCXT8E01++oUaMlwiETi9PIvFIYcyiJaEIihn+e7hyMjRPq4O8sjXeuKw==")) ; requires the "aes" egg
(compression deflate) ; requires the "z3" egg

;; Simple example to include only specific directories from /etc and
;; your homedir (the default is to include all files)
(rule "/"
      (not (or (name "home")
               (name "etc")))
      exclude)
(rule "/home/user"
      (not (or (name "docs")
               (name ".emacs")))
      exclude)

You will probably have already noticed the other file you'll have to create, which is referenced in the Ugarit config. This contains the S3-specific settings. This file is separated from the ugarit config, because the backend runs as a separate process, and we don't want to pass the secrets as a command line argument (that would expose it to prying eyes via ps).

Here is what it should look like:

(access-key "ABCDEFGHIJ0123456789")
(secret-key "my-secret-password")
(bucket "my-ugarit-bucket")
;; Optional, to be used if you have a different
;; storage provider than Amazon:
(base-hostname "o.auroraobjects.eu")

Finally, once you have set it up, you can make a backup like usual:

 # ugarit snapshot /etc/ugarit/host.conf my-host /

That's it! You can of course explore the snapshot just as easily:

 $ ugarit explore /etc/ugarit/host.conf

This will still perform the sudo command to run the backend, so this may ask you for your sudo password. You can also run it as root (which is recommended when you need to restore something, so the permissions can be set correctly).

Configuration options

The following configuration options are supported in the configuration file that's specific to the S3 backend.

The following options are all required:

 (access-key STRING)
 (secret-key STRING)
 (bucket STRING)

These are the credentials to use when connecting to your S3 store and the bucket in which the data is kept. Note that the bucket must already exist before you can make a snapshot!

The following options are optional:

 (base-hostname STRING)

The base hostname to use. Defaults to "s3.amazonaws.com". By default, the bucket name will be prepended to this host name, separated with a dot.

 use-bucket-path

When this occurs in the file (without parentheses), the bucket is not prepended to the base-hostname, but it is used as the first component of the path (for compatibility with certain kinds of Ceph setups).

Security recommendations

Encrypt your data

Rule number 1: When backing up to a cloud service like S3, always use encryption (e.g. the aes option in your ugarit config)!

You never know when the provider may get hacked, old disks might get discarded and scooped up, your buckets might accidentally be world-readable (a bigger problem than you might think) or some agency might be poking its nose where it doesn't belong.

Salt your data

As per the Ugarit documentation, use the (hash) option to provide a salt, so that blocks with the same content can't be easily detected across different vaults. The only disadvantage of this is that this will use more disk space if you use the same S3 bucket for multiple hosts - something which is explicity not recommended.

If you use multiple vaults on the same host, feel free to use the same salt between those vaults to allow sharing of identical blocks.

Use separate S3 accounts for each machine

In a worst case scenario, your machine might get hacked. If this happens, the attacker might find your Ugarit configuration, which includes S3 credentials. Using these credentials, the attacker might decide to wipe or overwrite/corrupt all data from your S3 bucket.

That's why it's important to use a different S3 account for every machine. If one gets hacked, the backups from the other machines cannot also be corrupted.

There is nothing Ugarit can do against this: if you back up to a directory, that can also be erased.

To mitigate this, you could enable S3 versioning to make it behave as an append-only file system, but note that that will also drive up your price, and the current backend has no support for reading from older S3 versions. Neither does s3cmd, so you'll need to determine if it's worthwhile.

Use privilege separation for the backend

Using sudo (or doas, etc) to run the backend is strongly recommended. This will avoid having to make network connections as root. Use an account that has as few privileges as possible; it will only need access to the S3 configuration file, the OpenSSL default certificate store, and to make network connections.

Do not use sensitive info for your tag names

Tag names are used in the (unencrypted) file names in the backing store, so you should avoid using tag names that might be considered sensitive information.

Note that this goes for every backend, as this is part of the Ugarit design.

Authors

Peter Bex

Changelog

Repository

http://code.more-magic.net/ugarit-backend-s3

License

 All code in this egg is explicitly placed in the public domain. You may do whatever you want with it.