Community Post

Building an Amazon S3 API with Express and Multer-S3

Charles Nnaji

Introduction

Hi guys, this is my first tutorial on http://www.scotch.io and i am glad to be here today. I found out that most programmatic problems are not yet solved online and i just want to write good content on solving programmatic problems and sharing my knowledge and skills to the world.

I was working on a project recently that enables users to store files on their Amazon S3 buckets, i was able to couple some api's with express and integrate Amazon with Multer-S3 for file uploads to buckets.

Amazon S3

Amazon Simple Storage Service (Amazon S3), provides developers and IT teams with secure, durable, highly-scalable cloud storage. Amazon S3 is easy to use object storage, with a simple web service interface to store and retrieve any amount of data from anywhere on the web. With Amazon S3, you pay only for the storage you actually use. There is no minimum fee and no setup cost. To learn more about Amazon S3 check out http://www.aws.amazon.com/s3. It is free, cheap and easy.

Prerequisites

  1. Node js
  2. Express
  3. Multer and Multer S3 (npm install multer multer-s3)
  4. Aws sdk (npm install aws-sdk)

Installations

npm install multer multer-s3 aws-sdk

Let the coding begin!

Before you start coding, you first need to register for a 12 months trial for Amazon s3. Once you have registered an account, you need to create a config file to store you s3 credentials. There are different ways to store your credentials on Amazon s3, check out http://docs.aws.amazon.com/AWSJavaScriptSDK/guide/node-configuring.html.

But, i used the fourth configuration method which is loading your credentials on a JSON file on disk. First create a file named s3_config.json and save to your project folder and save your credentials to the file as shown below.

 {
    "accessKeyId": "access key id goes here",
    "secretAccessKey": "secret access key goes here",
    "region": "us-east-1"
}

Note: To get you access key and your secret access key, you need to have an aws account with Amazon. Check out this link to create your credentials http://www.cloudberrylab.com/blog/how-to-find-your-aws-access-key-id-and-secret-access-key-and-register-with-cloudberry-s3-explorer/.

Creating the Api

To create the api is simple and straight forward, just follow the example below and i am sure you already know how to setup an express app. If you already do, good job but if you don't i suggest you google your way out. Let's get to business!

var multer  = require('multer'),
    multerS3 = require('multer-s3'),
    fs = require('fs'),
    AWS = require('aws-sdk');

AWS.config.loadFromPath('./s3_config.json');
var s3 = new AWS.S3();

//Create bucket. Note: bucket name must be unique.
//Requires only bucketName via post 
//check [http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#createBucket-property](http://) for more info
exports.createBucket = function (req, res) {
    var item = req.body;
    var params = { Bucket: item.bucketName };
    s3.createBucket(params, function (err, data) {
        if (err) {
            return res.send({ "error": err });
        }
        res.send({ data });
    });
}

//List all buckets owned by the authenticate sender of the request. Note: bucket name must be unique.
//check http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#listBuckets-property for more info
exports.listBuckets = function (req, res) {
    s3.listBuckets({}, function (err, data) {
        if (err) {
            return res.send({ "error": err });
        }
        res.send({ data });
    });
}

//Delete bucket.
//Require bucketName via delete 
//check http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#deleteBucket-property for more info
exports.deleteBucket = function (req, res) {
    var item = req.body;
    var params = { Bucket: item.bucketName };
    s3.deleteBucket(params, function (err, data) {
        if (err) {
            return res.send({ "error": err });
        }
        res.send({ data });
    });
}

//Delete bucket cors configuration. 
// Requires bucketName via delete
//check http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#deleteBucketCors-property for more info
exports.deleteBucketCors = function (req, res) {
    var item = req.body;
    var params = { Bucket: item.bucketName };
    s3.deleteBucketCors(params, function (err, data) {
        if (err) {
            return res.send({ "error": err });
        }
        res.send({ data });
    });
}

//Retrieves objects from Amazon s3
//check http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#getObject-property to configure params properties
//eg var params = {Bucket: 'bucketname', Key:'keyname'}
exports.getObjects = function (req, res) {
    var item = req.body;
    var params = { Bucket: req.params.bucketName };
    s3.getObject(params, function (err, data) {
        if (err) {
            return res.send({ "error": err });
        }
        res.send({ data });
    });
}

//Delete qn object
//check http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#deleteObject-property for more info
exports.deleteObject = function (req, res) {
    var item = req.body;
    var params = { Bucket: item.bucketName, Key: item.key };
    s3.deleteObjects(params, function (err, data) {
        if (err) {
            return res.send({ "error": err });
        }
        res.send({ data });
    });
}

//cloud image uploader using multer-s3 
//Pass the bucket name to the bucketName param to upload the file to the bucket 
exports.uploadFile = function (req, res) {
    var item = req.body;
    var upload = multer({
        storage: multerS3({
            s3: s3,
            bucket: item.bucketName,
            metadata: function (req, file, cb) {
                cb(null, { fieldName: file.fieldname });
            },
            key: function (req, file, cb) {
                cb(null, Date.now().toString())
            }
        })
    })
}

Conclusions

In this tutorial we have seen how aws s3 helps to integrate a simple cloud storage mechanism and you have also learnt how to configure your express app to enable s3 storage.

We covered all the basic, but i did not include any git repo, don't worry yourself much, just the code i provided can get you started.

As this is my first tutorial on http://www.scotch.io, i hope to release more rich tutorials in the future that solves your needs.

Charles Nnaji

My journey into the IT world started in the 2000s, i started with Microsoft technologies which made me a good programmer and it also improved my coding skills and ability to learn other languages. I love what i do. Steve Jobs once said that everybody should learn how to code, because it makes you think. I have learnt a lot and i would love to share my ideas and skills to the world. Coding makes me think and tackle most problems in life.