Amazon S3 + s3backer on Ubuntu 12.04 mini-HOWTO
Submitted by hswong3i on Fri, 2012-12-14 00:08
Amazon S3
provide object storage with "highly scalable, reliable, secure, fast,
inexpensive infrastructure" which very useful for different use cases,
e.g. storing your team shared files, your website backup, etc. BTW, it
is using different protocal when compare with FTP/FUSE/NFS/iSCSI/etc so
we need some trick in order to use it as like as local file system and
mount it as usual.
s3backer give you an alternative answer when compare with S3QL: s3backer just give you a block device for loopback mount on top of FUSE, so you can format it as whatever EXT4/LVM/OCFS2/etc (where S3QL give you entire file system which it implement internally); s3backer also support other object storage service provider, e.g. Google Storage, Amazon S3 or OpenStack, so we can switch to different vendor if do required; simple implementation so more stable; and key point is: very fast when compare with other solution!
This mini-HOWTO will guide you though some basic for mounting your S3 bucket as local file system, so you can use it for whatever purpose. You need to have Ubuntu 12.04 LTS install correctly.
On the AWS page, hover over the "Your Account" tab, and select "Security Credentials". Find the section labeled "Your access keys" - make a note of your Access Key ID, then click on the link labeled "Show" under "Secret Access Keys" and note that information, too. You will have to provide both pieces of information to the s3 commands you'll be using later.
s3backer give you an alternative answer when compare with S3QL: s3backer just give you a block device for loopback mount on top of FUSE, so you can format it as whatever EXT4/LVM/OCFS2/etc (where S3QL give you entire file system which it implement internally); s3backer also support other object storage service provider, e.g. Google Storage, Amazon S3 or OpenStack, so we can switch to different vendor if do required; simple implementation so more stable; and key point is: very fast when compare with other solution!
This mini-HOWTO will guide you though some basic for mounting your S3 bucket as local file system, so you can use it for whatever purpose. You need to have Ubuntu 12.04 LTS install correctly.
Install s3cmd and s3backer
Just simply execute following commands (I use uuid to create a unique S3 bucket name):aptitude update && \ aptitude -y install s3cmd subversion uuid && \ aptitude -y install libcurl4-openssl-dev libfuse-dev libexpat1-dev
If you are using Ubuntu 12.04 LTS, I would also recommend you to install the latest version of supported kernel, so entitle the benefit of remove unused block automatically (i.e. reduce your S3 cost)Configure FUSE so allow other user to use the block device:
aptitude -y install linux-generic-lts-raring
sed -i 's/^#\s*\(user_allow_other\)/\1/g' /etc/fuse.confNow download the s3backer source package, build and install it:
cd /usr/local/src svn checkout http://s3backer.googlecode.com/svn/trunk/ s3backer-read-only cd s3backer-read-only ./autogen.sh ./configure make make install
Get Your AWS Account Information
If you haven't already, sign up with Amazon Web Services and enable S3 for your account.On the AWS page, hover over the "Your Account" tab, and select "Security Credentials". Find the section labeled "Your access keys" - make a note of your Access Key ID, then click on the link labeled "Show" under "Secret Access Keys" and note that information, too. You will have to provide both pieces of information to the s3 commands you'll be using later.
Setup s3cmd and Create a Bucket
S3 lets you create buckets to store your information; you have to use the AWS API to create buckets, and you won't be able to create buckets with s3fs. If you haven't already created a bucket in your S3 account, you can use the s3cmd program to set one up (Just answer the qustion and fill up with your information):s3cmd --configureOnce complete s3cmd should show your some message similar as below:
Test access with supplied credentials? [Y/n] Please wait... Success. Your access key and secret key worked fine :-) Now verifying that encryption works... Success. Encryption and decryption worked fine :-) Save settings? [y/N] y Configuration saved to '/root/.s3cfg'First of all let's generate an unique bucket name with uuid, which we will reuse in on going example:
export set BUCKET=`uuid`Now you can create your bucket with:
s3cmd mb --bucket-location=ap-southeast-1 s3://$BUCKETWill prompt you with a result bucket name as below:
Bucket 's3://915cfdf8-44ca-11e2-9759-12314000610e/' createdTest putting some file to it:
s3cmd put /var/log/syslog s3://$BUCKET/And list object to verify if all works well:
s3cmd ls s3://$BUCKET/
Configure s3backer Authentication, Initialize and Test Mount Manually
Now we go to s3backer. First of all create an authentication file, e.g. /root/.s3backer_passwd in following format:accessID:accessKeyUpdate /root/.s3backer_passwd permission:
chmod 600 /root/.s3backer_passwdCreate the target folder for s3backer:
mkdir -p /backup mkdir -p /s3backer/$BUCKETInitialize the bucket for s3backer:
s3backer \ --blockSize=1M \ --listBlocks \ --size=1T \ --vhost \ $BUCKET \ /s3backer/$BUCKET
If you hope to have a encrypted file system, a lazy format as below:Format the target block device as EXT4:
s3backer \ --accessFile=/root/.s3backer_passwd \ --blockSize=1M \ --compress \ --encrypt \ --listBlocks \ --passwordFile=/root/.s3backer_passwd \ --size=1T \ --vhost \ $BUCKET \ /s3backer/$BUCKET
mkfs.ext4 -b 4096 -F /s3backer/$BUCKET/fileThen mount it as loopback device:
mount -o loop /s3backer/$BUCKET/file /backupFor sure you should put some file to it, umount and mount it again and again, also check with df -h ;-)
Setup Automatic Mounting
Automatic mounting for s3backer is extremely simple when compare with S3QL: we can handle it within /etc/fstab! Something similar as below:cat >> /etc/fstab <<- span="">EOF s3backer#$BUCKET /s3backer/$BUCKET fuse _netdev,defaults,noauto,blockSize=1M,listBlocks,size=1T,vhost 0 0 /s3backer/$BUCKET/file /backup ext4 _netdev,defaults,noauto,relatime 0 0 EOF->
For kernel > 3.7 with TRIM enabled version:
cat >> /etc/fstab <<- span="">EOF s3backer#$BUCKET /s3backer/$BUCKET fuse _netdev,defaults,noauto,blockSize=1M,listBlocks,size=1T,vhost 0 0 /s3backer/$BUCKET/file /backup ext4 _netdev,defaults,noauto,relatime,discard 0 0 EOF->
For kernel > 3.7 with TRIM enabled + encrpted version:Now you can mount them as usual:
cat >> /etc/fstab <<- span="">EOF s3backer#$BUCKET /s3backer/$BUCKET fuse _netdev,defaults,noauto,accessFile=/root/.s3backer_passwd,blockSize=1M,compress,encrypt,listBlocks,passwordFile=/root/.s3backer_passwd,size=1T,vhost 0 0 /s3backer/$BUCKET/file /backup ext4 _netdev,defaults,noauto,relatime,discard 0 0 EOF->
mount /s3backer/$BUCKET mount /backupAfter reboot it will also mount automatically ;-)
Reference
- http://code.google.com/p/s3backer/wiki/BuildAndInstall
- http://code.google.com/p/s3backer/wiki/RunningTheDemo
- http://code.google.com/p/s3backer/wiki/CreatingANewFilesystem
- https://groups.google.com/forum/#!topic/s3backer-devel/myzw2sCrCkY
No comments:
Post a Comment