And DeepStorage Labs
As recently as two or three years ago I would only recommend object storage to users who had requirements exceeding the capabilities of an enterprise NAS file system. The additional effort of writing applications to use new, and different, APIs wasn’t worth it for less than hundreds of terabytes of data.
Times have changed. Backup, archiving, and other data-driven applications are supporting object storage for their repositories.
Developers who in 2010 refused to learn the S3 API now want to save objects to make their apps portable to the public cloud, or just because that’s the more modern way to do things.
Both of these trends make smaller object storage servers attractive as backup targets and development environments.
Scality’s open source S3 Server is a good fit for applications that just can’t justify the 3-5 node practical minimums for full-scale object stores like Scality’s Ring. The S3 Server runs in a Docker container, which simplifies both distribution and installation.
Running an S3 Server container creates a stand-alone object store. For scale-out, erasure coding, encryption, and the like, you will have to buy Scality RING, because the folks at Scality know better than to give everything away.
Running S3 Server was going to be my first hands-on adventure with containers, and therefore container storage, and I was excited to get started.
For my first S3 Server install I followed Scality’s simple getting started page. I installed S3 Server in a Windows VM and was up and running in just a few minutes. The recommended install used Docker Toolbox to install S3 Server into a Virtual Box VM on my Windows machine.
Once the S3 Server VM was up and running, I could get and put objects with CyberDuck from Windows. If I were a developer, I might be happy with just a local object store. Being a storage guy, I wanted to perform some more substantial testing, but the S3 Server was running on a 192.168 address provided by Virtual Box.
To access my shiny new object store from some other system I’d have to spend some time researching Virtual Box networking and update the lab’s routers to include the new subnet. I decided I’d rather spend my time researching S3 Server than Virtual Box networking.
Now that I knew that S3 Server basically worked, I spun up a CentOS 7 VM, installed Docker with YUM, and installed S3 Server with sudodocker pull Scality/s3server. After a few minutes of downloading and installing, I started up an S3 Server container with sudodocker run -d –name s3server -p 8085:8085 scality/s3server and had an object store I could use for some real work.
I backed up my laptop using Cloudberry Backup. The 38.5GB job completed in 2:12. That’s a rate of 4.3MBps, which is about 10 times the rate of my backups to Amazon S3. Granted, I have a gigabit link to the S3 Server and only a 20Mbps Internet connection (living in Santa Fe has its advantages, but fast Internet service isn’t one of them) but my backups to Amazon S3 backups don’t even saturate that.
By default, S3 Server stores its data within the container, persisting it over container restarts. However, for a backup target or other significant applications I want a bit more control over where my data goes. So I posted a series of questions to the S3 Server support forum asking how I controlled where S3 Server stored its data. The response, which came from a Scality employee just a few hours later, informed me that 3 Server uses Docker Volumes, giving users a wide variety of options for storage.
Unlike scale-out storage systems that distribute data across their nodes for protection, S3 Server relies on lower storage layers for protection. I mounted a virtual disk from the lab’s Tintri T540 hybrid storage system and another from a local RAID-6 set of 7200RPM disks. It was then simple to direct the S3 Server’s metadata to the hybrid drive and the data to the spinning disks. We’ll be using this S3 Server as the lab backup target and will report any significant events here in a later post
All in all, S3 Server makes it easy to create an object store when a full blown, scale-out system like Scality’s RING would be too much. Open source, written in JSON, and running in Docker containers S3 Server seems the perfect solution for private developer repositories, backup repositories and other applications that don’t need a full blown scale-out storage system.
There are a few rough edges; for instance, changing passwords and installing certificates requires editing JSON files. But even for a Linux novice like me, the full setup of a good-size object store took less than an hour.
As open source software S3 Server is free to use. For production use; Scality offers enterprise support for $950/month covering an unlimited number of S3 Servers.
This post was sponsored by Scality.