Subscribe to Lumen Cloud Knowledge BaseDescription
Lumen Cloud Object Storage is an ideal repository for unstructured data ranging from media files to database backups. The Object Storage service is Amazon S3 compatible which means that code and tools that work with Amazon S3 should work seamlessly with Lumen Cloud Object Storage. This KB article shows you how to use the raw REST API and the AWS SDK (for .NET and Node.js) to interact with Object Storage.
Audience
- Developers
Prerequisites
- Have users and buckets created in Object Storage.
- See the KB article Using Object Storage from the Control Portal for a walkthrough of users and buckets.
Using the Object Storage REST API from .NET
The Lumen Cloud Object Storage service offers an Amazon S3-compatible web services endpoint. The endpoint has the same authentication, resources, and payloads as defined in the Amazon S3 documentation. The steps below show how to consume Object Storage from a custom .NET application. Note that the source code for this sample application can be downloaded from GitHub.
Detailed Steps
Log into the Control Portal to acquire your user credentials.
-
From the Navigation Menu, Click Services > Object Storage. On the Object Storage page, click the Users tab.
-
Click the user name associated with the buckets (or has access to an individual bucket) that you want to access. Locate and record the access key id and the secret access key values. These act as your credentials for all API operations.
-
Create a new Visual Studio project. In this example, the project is a Console application.
-
Define a new "helper" function that can generate the proper authentication header that the service expects. The function below accepts values used to generate the authentication string, and is encoded and signed.
///
/// Helper function to generate the required authentication header for messages to S3 endpoint
///
private static string GenerateApiAuthHeader(string verb, string timestamp, string contentType, string resource, string secret)
{
string authHeader = verb + "\n" + //HTTP verb
"\n" + //content-md5
contentType + "\n" + //conten-type
"\n" + //date
"x-amz-date:" + timestamp + "\n" + //optionally, AMZ headers
resource; //resource
Encoding encoding = new UTF8Encoding();
HMACSHA1 signature = new HMACSHA1();
signature.Key = encoding.GetBytes(secret);
byte[] bytes = signature.ComputeHash(encoding.GetBytes(authHeader));
return Convert.ToBase64String(bytes);
}
- Define variables to hold the authentication credentials for Object Storage. In the example below, these values are stored in the application's configuration file.
private static string adminAccessKey = System.Configuration.ConfigurationManager.AppSettings["ObjectStorageKey"];
private static string adminAccessSecret = System.Configuration.ConfigurationManager.AppSettings["ObjectStorageSecret"];
- Write the code that calls the Object Storage endpoint and requests a list of all buckets. The code below sets the URL for Object Storage (which is based on which region you are querying; this URL can be seem by looking at the settings for one of your buckets). The authorization header requires a timestamp (as authentication is only valid for a certain period) and this value is formatted in the manner expected by the endpoint. After the authentication header is generated by our helper function, the appropriate headers are added to the request and an HTTP GET request is issued against the Object Storage URL. The response is retrieved as a string and loaded into an XML object for formatting.
//set up variables used by this method
string s3Url = "https://canada.os.ctl.io/";
XDocument resultDoc = new XDocument();
string timestamp = String.Format("{0:r}", DateTime.UtcNow); //need UtcNow, not just Now or get wrong time
//call helper method to generate correct header
string authHeader = GenerateApiAuthHeader("GET", timestamp, string.Empty, "/", adminAccessSecret);
//make RESTful call to S3-compliant endpoint
using (var client = new HttpClient())
{
HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Get, s3Url);
//add token to header
request.Headers.Add("Authorization", "AWS " + adminAccessKey + ":" + authHeader);
request.Headers.Add("x-amz-date", timestamp);
var response = client.SendAsync(request);
if (response.Result.IsSuccessStatusCode)
{
//synchronous call
var responseContent = response.Result.Content;
//synchronous read
string responseString = responseContent.ReadAsStringAsync().Result;
//throw into XDocument to get nice XML formatting
resultDoc = XDocument.Parse(responseString);
Console.WriteLine(resultDoc.ToString());
}
else
{
//error
Console.WriteLine("ERROR: " + response.Result.StatusCode.ToString());
}
}
- Add code that queries Object Storage for the objects contained within an individual bucket. This is slightly different than the previous action in that the resource name (in this case, the bucket name) is added to the URL and passed into the helper method.
Console.WriteLine("ACTION: Enter the name of a bucket to open: ");
string inputbucket = Console.ReadLine();
//url updated to include the resource (bucket) value
s3Url = "https://canada.os.ctl.io/" + inputbucket;
timestamp = String.Format("{0:r}", DateTime.UtcNow); //need UtcNow, not just Now or get wrong time
//call helper method to generate correct header
authHeader = GenerateApiAuthHeader("GET", timestamp, string.Empty, "/" + inputbucket, adminAccessSecret);
//make RESTful call to S3-compliant endpoint
using (var client2 = new HttpClient())
{
HttpRequestMessage request2 = new HttpRequestMessage(HttpMethod.Get, s3Url);
//add token to header
request2.Headers.Add("Authorization", "AWS " + adminAccessKey + ":" + authHeader);
request2.Headers.Add("x-amz-date", timestamp);
var response2 = client2.SendAsync(request2);
if (response2.Result.IsSuccessStatusCode)
{
//synchronous call
var responseContent2 = response2.Result.Content;
//synchronous read
string responseString2 = responseContent2.ReadAsStringAsync().Result;
//throw into XDocument to get nice XML formatting
resultDoc = XDocument.Parse(responseString2);
Console.WriteLine(resultDoc.ToString());
}
else
{
//error
Console.WriteLine("ERROR: " + response2.Result.StatusCode.ToString());
}
Console.ReadLine();
}
-
Run the application and first see that all Object Storage buckets are returned, and then ask for the contents of a single bucket.
Using Amazon SDK for .NET to Interact with Object Storage
The Amazon Web Services team created and maintain a set of SDKs for developers to use when interacting with their cloud platform. Because Lumen Cloud Object Storage is S3-compatible, developers can reuse these existing SDKs when interacting with Object Storage. An SDK is typically MUCH easier to work with than a raw API as much of the complexity is hidden. Note that the source code for this sample application can be downloaded from GitHub.
Detailed Steps
-
Download the AWS SDK for .NET and install it. Or, open Visual Studio and install the SDK as part of a NuGet package. To install the NuGet package, right click the Visual Studio project, choose "Manage NuGet Packages", search for "AWS" and install the package into this project.
-
Add code that uses the AWS SDK to read the list of buckets from Object Storage. First, a configuration object is instantiated and the Lumen Cloud Object Storage URL is provided. Then, the client object is created and provided the credentials and configuration. The SDK uses strong typing, so the client object specifically calls "ListBuckets" and gets back a list of buckets with properties such as "BucketName."
//create configuration that points to different URL
AmazonS3Config config = new AmazonS3Config()
{
ServiceURL = "canada.os.ctl.io"
};
AmazonS3Client client = new AmazonS3Client(adminAccessKey, adminAccessSecret, config);
/*
* List buckets
*/
Console.WriteLine("ACTION: List all the buckets");
ListBucketsResponse resp = client.ListBuckets();
foreach (S3Bucket bucket in resp.Buckets)
{
Console.WriteLine("-" + bucket.BucketName);
}
- Add a second action that queries the contents within a given bucket. This code reuses the client object created earlier, and calls the "ListObjects" operation to get back a list of objects.
/*
* List objects in a single bucket
*/
Console.WriteLine("ACTION: Enter the name of a bucket to open: ");
string inputbucket = Console.ReadLine();
ListObjectsRequest objReq = new ListObjectsRequest() { BucketName = inputbucket };
ListObjectsResponse objResp = client.ListObjects(objReq);
foreach (S3Object obj in objResp.S3Objects)
{
Console.WriteLine("-" + obj.Key);
}
- Run the application and see the list of buckets and then the list of objects in a selected bucket.
Using the Amazon SDK for Node.js to Interact with Object Storage
Node.js is a popular platform for building high-performing JavaScript-friendly applications. Amazon Web Services also ships an SDK for Node.js, and that SDK works perfectly with Lumen Cloud Object Storage. Note that the source code for this sample application can be downloaded from GitHub.
Detailed Steps
-
Create a new Node.js project. This example solution uses the Express framework to provide an MVC foundation to the application.
-
Use the Node Package Manager (npm) to install the AWS SDK for Node by using the
npm install aws-sdkcommand. -
Create a credentials.json file that contains the Object Storage credentials and region.
{
"accessKeyId": "KEYID",
"secretAccessKey": "SECRET",
"region":"Canada"
}
- In the route (controller), add the code to respond to the page load event. This code queries Object Storage via the AWS SDK for Node. See that it loads the credentials file, and passes the Object Storage URL and signature version into the object constructor. The listBuckets operation returns an array of buckets. Here, that array is returned to the view.
var AWS = require('aws-sdk');
exports.index = function(req, res){
AWS.config.loadFromPath('./credentials.json');
var s3 = new AWS.S3({endpoint:'https://canada.os.ctl.io', signatureVersion:'v2'});
s3.listBuckets(params = {}, function(err, data){
console.log(data.Buckets);
res.render('index', { title: 'Lumen Cloud Object Storage Bucket List', buckets: data.Buckets });
});
};
- On the Express view, the code loops through each bucket and prints the name of the bucket in a table cell.
table(class="table table-striped table-bordered table-hover")
thead
tr
th Bucket Name
tbody
each bucket in buckets
tr
td.cell
| #{bucket.Name}
- Run the Node.js application and see that the list of Lumen Cloud Object Storage buckets is retrieved and displayed on the page.
Using the Amazon SDK for Java to Interact with Object Storage
Amazon Web Services also ships an SDK for Java, and that SDK works with Lumen Cloud Object Storage. To get started, include the package in your project; an example for Gradle is shown below (syntax can vary if a different build system is being used and/or if a different version of the package is desired).
compile 'com.amazonaws:aws-java-sdk-s3:1.11.411'
You can then use the following code to create a client object that can be used to interact with Lumen Cloud Object Storage:
AmazonS3 s3client = AmazonS3ClientBuilder.standard()
.withClientConfiguration(new ClientConfiguration().withSignerOverride("S3SignerType"))
.withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration("https://canada.os.ctl.io", ""))
.build()
There are several different options to configure the client's behavior, including how to supply it with your credentials. Documentation on how to use the client to perform operations like listing buckets, listing objects in a bucket, creating a bucket, creating an object in a bucket, and more can be found here.
