Object Storage needs to be FAST now

Cloudian HyperStore is able to meet the demands of modern applications with a flash based Object Storage offering.

Henry GolasHenry Golas, Director of Technology, Cloudian

View LinkedIn Profile

Historically, Object Storage has been viewed as a cheap and cheerful archive for large amounts of data. These archives were the ‘last stop for data’, data that needed to be kept for long periods of time, but not accessed very frequently. The primary focus was storing data as cost effectively as possible, without much regard for other requirements.

As applications transition to a cloud native world, the requirements for data storage have changed. Data needs to be available, accessible via S3 API, and performant. This shift has moved Object Storage from a “Dollars per GB” to a “Dollars per IOPs” model as today’s requirements now justify flash media for Object Storage platforms. 

Three use cases pushing the performance requirements of object storage:

1) By leveraging the power of custom metadata, Object Storage is gaining momentum with applications that store transactional data in object format. As an example, imagine a financial application that bundles all the relevant supporting documentation for a transaction, like a mortgage, with the object and adds searchable metadata. Downstream applications would be searching and reading the same objects. Flash based Object Storage is required to keep up with the transactional throughput of the application.

2) Data streaming and Internet of Things (IOT) now generate Terabytes and Petabytes of data per hour. Using Autonomous Vehicles as an example, daily data generation estimates are from 3TB-15TB per day, per vehicle. Testing and training a fleet of AVs now requires flash based Object Storage to keep up with the data volume.

3) Data analytics and big data players are extending their data lakes with Object Storage. This allows the separation of compute and storage resources. Why buy compute when storage is required? Hadoop supports S3A requests for S3 compatible object storage, allowing for direct read and writes of S3 objects. Companies like Vertica (https://www.vertica.com/) and their Vertical Analytics Platform support S3 for communal storage, allowing compute resources to be scaled independently. Data analytics and big data need performant object storage to keep up with modern demands.

 


The net here is that modern applications aren’t just storing data and forgetting about it. These applications are actively using Object Storage as a performant storage platform. 

Cloudian HyperStore is able to meet the demands of modern applications with a flash based Object Storage offering. Cloudian delivers this software defined storage solution as a turnkey Cloudian appliance or on standard x86 hardware.

To further increase performance and drive cost efficiencies, Cloudian HyperStore’s data tiering functionality allows a smaller subset of hot data to reside on flash object storage, while the remaining cold data set stays on traditional HDD technology.

 

 

This tiered solution blends the traditional cost centric “Dollars per GB” approach, with the “Dollars per IOPs” requirements of modern applications, allowing up to 40% cost reductions vs an all flash-based solution.

 

Check out more details at cloudian.com/flash

Do You Know Where Your Metadata Is? You Should.

There is a lot of talk in object storage about separating metadata from user data. How metadata is stored makes a critical difference in the success of your storage project. When you understand the benefits of a dedicated metadata store, you quickly realize why Cloudian’s HyperStore object storage platform has one.

Henry GolasHenry Golas, Director of Technology, Cloudian

View LinkedIn Profile

There is a lot of talk in object storage about separating metadata from user data. How metadata is stored makes a critical difference in the success of your storage project. When you understand the benefits of a dedicated metadata store, you quickly realize why Cloudian’s HyperStore object storage platform has one.

binge watchWhen we talk about data and data storage today, the conversations are typically about data at scale—petabytes or even exabytes. To put this in context, if you binge-watched Netflix, non-stop, for about 3.5 years, you would have watched a petabyte worth of data. Now, an exabyte would take about 3,584 years to watch.

So, what does metadata have to do with this?

Metadata is information about data. There is system metadata that contains items like date and system information about the data, and there is custom metadata, which provides context for data. For example, for the 1993 movie Groundhog Day, custom metadata would include words like “1993, Bill Murray, romantic comedy, classics, feel-good.”

With that context, here are 4 reasons why having a dedicated metadata store is so important:

1. Storing metadata in a dedicated store—i.e., a database—makes data management and metadata operations easier without impacting system performance. If you don’t have a centralized store, a data request, such as an object search or bucket listing, will have to traverse the entire object store, which can contain billions of objects and be 100s of PBs in size, every single time.

See the difference below? On the left, I’m scrolling titles based on metadata, and when I find Groundhog Day, I can simply click it and start watching. On the right, without the help of metadata, I must start searching at the front of the row and keep going until I find my movie.

groundhog day

2. Keeping custom metadata in a metadata store allows for advanced analytics and searches, with the ability to search on virtually any piece of metadata, be it custom or system metadata.

Continuing with the movie example, if I wanted to list all the movies from 1993, it’s a simple metadata query. Without a metadata store, I’d have to scan each title and compile a list. Now, imagine a real-world analytics example with millions or billions of objects. Because of the sheer volume, scanning each object is not a realistic option.

romatic comedy

metadata store3. By having an API-accessible metadata store, search and analytics engines, such as ELK Stack, can integrate seamlessly. Search and analytics operations can therefore occur without impacting data service.

While this type of integration is possible without a metadata store, there are significant performance risks by having to scan the system for each query (see point #1 above).

4. Lastly, by definition, traversing data and metadata in search of metadata means traversing actual data. This could be a security concern for sensitive or restricted data environments: PII, PCI, Healthcare & HIPPA compliance, etc. Having a dedicated metadata store removes this concern.

By now, it should be clear that enterprise object storage platforms need dedicated metadata stores.

Cloudian’s HyperStore object storage platform leverages a dedicated metadata store, enabling metadata transactions to occur without impacting system performance or data services. Cloudian HyperStore also allows API calls for search and analytics engines such as ELK Stack. Lastly, Cloudian HyperStore provides industry-leading, government-certified data security, including protection against ransomware and unauthorized root access.

Learn more at cloudian.com/products/hyperstore/

Enhancing Object Storage Analytics: Adding Metadata Labels to S3 Images with TensorFlow

Object storage is known for its scalability and easy-to-use S3 APIs, but to make that object data useful for analytics, metadata about the objects sometimes needs to be added.  This article describes a case study of adding and then using metadata of S3 objects.  Starting with images stored in Cloudian HyperStore object storage, we use a TensorFlow machine learning model to identify what’s depicted in the image, then attach those labels to each image as S3 metadata, and finally automatically index and search the object metadata using ElasticSearch and Kibana.

tensorflow pod diagram

INPUT
Unlabeled images stored in HyperStore S3 bucket.

OUTPUT
Images with metadata of labels of what’s in the image stored back in HyperStore and ElasticSearch.

METHOD
Use a TensorFlow deep learning model to determine labels of what’s in the image.
Use HyperStore’s ElasticSearch plugin to make metadata searchable and visualizable.

In an S3 bucket named “images,” we upload about 300 images of common items, including animals, vehicles, and household goods.  Using an object store for a collection of images, it’s very convenient to store a large amount of data easily and economically.

images bucket

By locating the analytics/computation processing close to the data, HyperStore takes advantage of the data locality and an edge-hub topology for efficient and timely processing.  It’s fast, with processing as close as possible to where the data is generated; cheap, with minimal network transfer costs for an upload and subsequent downloads, and secure because the data can be kept private and protected.

The image recognition process reads each object from the S3 bucket and calculates the image classifications by applying the TensorFlow model.  The S3 list-objects API is used to iterate over each object in a bucket.  For each object, checks are first done, including confirming the Content-Type is an image and the size is not above a threshold.  The image is then scaled to a fixed size, and the model is executed based on the TensorFlow’s LabelImage class.  The TensorFlow model used for image recognition is the pre-trained Inception 5h that recognizes 1,000 classes of images from ImageNet.

Below are examples of input images and the resulting classification outputs as a label and associated probability after the image recognition process runs.

red fox
2925[main] INFO com.cloudian.hap.LabelImage images/fox.jpg:
red fox (58.52% likely)
kit fox (39.54% likely)
coyote (0.73% likely)
grey fox (0.71% likely)
red wolf (0.25% likely)

image meta data

5095 [main] INFO com.cloudian.hap.LabelImage images/iphone.jpeg:
cellular telephone (41.24% likely)
hand-held computer (40.34% likely)
pay-phone (7.52% likely)
iPod (3.88% likely)
remote control (1.54% likely)

Some configurations to control the classifier:

image classifier

The image labels and their associated probabilities are added to the object using S3 user-defined metadata where the key is the prefix “imgtag_” plus the label (e.g., “red fox”) and the value is the associated probability (e.g., “0.59”).  The label is URL-encoded to ASCII to conform to the metadata key requirements, notably the <SPACE> character is converted to ‘+’.  To update an existing object’s user-defined metadata, the S3 Copy Object API is used with the x-amz-metadata-directive: REPLACE header.  The object and its metadata are now stored in HyperStore S3.  This example with a S3 GET command on bucket “images” and object “fox.jpg” shows the user-defined metadata output:

metadata output
metadata summary

HyperStore has the capability of indexing object metadata in ElasticSearch.  Once in ElasticSearch, Kibana can be used for data exploration.

elasticsearch results

Here’s an example query to find all images where the label “kit fox” has probability greater than 0.4.  The Kibana query is bucketname:images AND userMetadata.imgtab_kit+fox>0.4 that returns 2 objects:

kibana 1

If you don’t care what type of “fox” it is, you can use wildcards in the Kibana query bucketname:images AND userMetadata.imgtag\*fox\*:* that returns 13 objects:

kibana 2

S3-compatible object stores like HyperStore have enabled storing PBs of data. These tools provide a convenient way to move the compute to the data and, as in this use case, to add metadata to the object data.  In the same spirit, we are developing more use cases to enhance object storage analytics, including processing streaming data and other machine learning tasks.

Categories