CenturyLink Cloud Guide to Access Object Storage as a Filesystem Using s3fs

Updated by Gavin Lai on Sep 14, 2016

Table of contents

Overview

s3fs is a FUSE filesystem that allows you to mount a S3 compatible bucket as a local filesystem. It stores files natively and transparently in S3 compatible Objective Storage (i.e., you can use other programs to access the same files). Maximum file size=64GB (as of this article, limited by s3fs, not CenturyLink Cloud Object Storage). The limitations of s3fs is listed here. For pricing of Object Storage, please see the pricing page.

Prerequisites

  • Access to the CenturyLink Cloud platform as an authorized user

  • Identify a CentOS Linux platform that you would like to have s3fs installed

  • Understanding the functions of Object Storage (beyond the scope of this article), to learn more, please see this knowledge article

  • Ability to install packages on the server

  • Access to CenturyLink Cloud Object Storage

Use Case Scenarios

As Object Storage is accessed using the Internet, it can be used in many scenarios. While mounting Object Storage as a Linux file system makes consuming Object Storage much easier, keep in mind that it is only accessed via the Internet, hence the speed and latency is not the same as local storage. With this in mind, using Object Storage in a file system can be a good alternative to expensive local storage. Use cases can be:

  • Data archiving

  • Backup

  • Internet speed data storage

  • Long term storage

If encryption is a requirement, please see s3QL, files created with S3QL can only be accessed using S3QL.

Preparation

In preparation. There are several factors need to be considered:

  • Deployment of s3fs will install Git, gcc along with Make and Fuse (for full list of packages, please refer to s3fs GitHub site)

  • Mountpoint for the Object Storage filesystem

Deployment

There are two methods to deploy s3fs in CenturyLink Cloud, it can be done manually or using a package.

Manual deployment

  1. s3fs deployment is done with the following commands on the CentOS 7 server:

    sudo yum install automake fuse fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel
    git clone https://github.com/s3fs-fuse/s3fs-fuse.git
    cd s3fs-fuse
    ./autogen.sh
    ./configure
    make
    sudo make install
    
  2. Once installed, access to the Object Storage screen in the Control Portal

    s3fs-objectstorage

  3. Locate the user and bucket for this deployment and record both the "Access Key ID" and "Secret Access Key"

  4. Create a password file with the "Access Key ID" and "Secret Access Key" and make proper permission on the file, for example:

    echo access_key_id:secret_access_key > /path_to_password_file/password_s3fs
    chmod 666 /path_to_password_file/password_s3fs
    
  5. Once the above steps are completed, create a directory for the mount point of the new filesystem

Execute using a package

A script (S3fs on CentOS 7) is created to speed up the deployment of s3fs on CenturyLink Cloud. The detail of the package is available here. The package can be install via the portal, API or CLI.

For API, please refer to the Execute Package API.

The CLI deployment can be done following the command in the CenturyLink Cloud Guide to CLI.

When using the portal, this Knowledge article

In the portal, the package can be found by searching for s3fs:

Search Window

The parameters required for this script are:

s3fs parameters

Testing

  • Test the installation with the command below (URL can be either useast.os.ctl.io or canada.os.ctl.io depending on the region of the bucket, for the current list of Object Storage end points, please see this knowledge article):

      s3fs mybucket:/ /path_to_mountpoint -o passwd_file=/path_to_password_file/password_s3fs -o url=https://canada.os.ctl.io/
    
  • If the command ran successfully, then a /etc/fstab entry can be inserted to mount the Object Storage bucket permanently, a fstab_entry file is created in the /root directory if the package is used (debug level set at info, it can be removed to minimized logging):

      s3fs#mybucket:/ /path_to_mountpoint fuse _netdev,dbglevel=info,allow_other,nodnscache,retries=5,url=https://canada.os.ctl.io/,passwd_file=/path_to_password_file/password_s3fs
    

    or

      mybucket:/ /path_to_mountpoint fuse.s3fs _netdev,dbglevel=info,allow_other,nodnscache,retries=5,url=https://canada.os.ctl.io/,passwd_file=/path_to_password_file/password_s3fs
    

Troubleshooting

  • Debug and run s3fs in the foreground can help during troubleshooting (-d x 2 to direct messages to STDOUT):

      s3fs mybucket:/ /path_to_mountpoint -o passwd_file=/path_to_password_file/password -o url=https://canada.os.ctl.io/ -d -d -f -o f2 -o curldbg
    

Support

Customer Support

Can’t find what you need?
Give us a call.

1.888.638.6771

M – F, 8am to 6pm