Data Observability with Elasticsearch and Cloudian

Challenges

Data growth and the need to extract timely insights from this increasingly strategic asset are two of the biggest challenges for enterprises today. Elasticsearch, the leading open source search and analytics platform, is used by enterprises of all sizes to index, search, and analyze their data, and to gain real-time insights for making data-driven business decisions.

The traditional Elasticsearch environments, built on compute-optimized servers with equally large and performant storage for all data are inefficient and get expensive very quickly. Storing all the data on performance-optimized storage drives up costs and makes it more difficult to accommodate growing data storage and analytics demands. Customers today demand a modern, cloud-like architecture that gives them the flexibility to choose and the flexibility to recognize cost efficiency from their hardware based on the way they analyze the data ideal resources for compute and storage and grow them independent of each other.

Modern Elasticsearch deployment with Cloudian HyperStore  

Elastics earch and Cloudian HyperStore, the industry’s leading on-prem S3-based object storage platform, seamlessly integrate to provide a modern, cloud-like architecture for on-prem enterprise customers. Searchable snapshots in Elastic let customers snapshot their infrequently accessed and read-only data and store it on the Cloudian HyperStore object storage platform. Searchable snapshots rely on the same snapshot mechanism you already use for backups and have minimal impact on your snapshot repository storage costs. They eliminate the need for replica shards, potentially halving the local storage needed to search your data. With Searchable Snapsots, customer have the option to implement cold or frozen data tiers, to reduce their storage and operating costs.

The cold tier offloads the replicas of the main data from expensive storage to the HyperStore tier. It maintains the same level of reliability and redundancy as your hot and warm tiers, with full support for automatic recovery from hardware failures on any of your nodes. This has potential to reduce warm tier storage cost by up to 50%.  With the frozen tier, all data is stored on scalable and economical S3-compatible HyperStore platform, with a cache layer to speed up the searches. This mode enables customers to reduce their overall storage infrastructure cost by up to 60%.    

Drop-in integration

Cloudian HyperStore, the industry leading on-prem S3 storage system, integrates out of the box with Elasticsearch’s searchable snapshot API’s. Only Cloudian offers a 100% native S3 API for objects, plus SMB/NFS for files. Proven in hundreds of enterprise deployments, HyperStore’s S3 API offers guaranteed compatibility with your S3-enabled applications.

Performance to handle the largest Elasticsearch environments 

Cloudian scales to petabytes with a scaling model that grows in both capacity and bandwidth. Cloudian’s streaming bandwidth provides for predictable backup windows. Data write bandwidth in excess of 5,000MB/s (or 18TB per hour) can be achieved. 

Exabyte-scalable 

Unlike conventional storage, Cloudian offers modular growth, letting you expand from terabytes to an exabyte without disruption. Embedded data redundancy features provide up to 14 nines data durability, removing the necessity of a separate data backup process. Compared with traditional enterprise storage — or with storage on compute-intensive servers — Cloudian saves up to 70% on TCO. 

Secure

To ensure data security, Cloudian provides AES-256 server-side encryption for data at rest, SSL for data in transit (HTTPS), role-based access controls, storage policies that can be applied at an object and bucket-level management granularity. In addition, the platform has DOD level security certifications including, FIPS 140-2, SEC Rule 17a-4(f), CFTC 17 C.F.R. § 1.31, FINRA 4511c, IDW PS 880 (German) and OR §§ 957ff (Swiss) regulations, and meets the data sanitization standards specified by NIST 800-88. 

Cloud-enabled

Cloudian is on-prem storage, but it integrates directly with public cloud storage services. Employ policy-based tools to replicate or tier data to AWS, GCP, or Azure for offsite DR, capacity expansion, or data analysis in the cloud. This built-in capability requires no additional software or licenses. 

Multi-Purpose Scalable Storage

Cloudian is the industry’s most compatible S3 API and integrates seamlessly with most S3-compatible applications. In addition to data protection for Elasticsearch environments, it can also provide scalable storage for data management applications from AWS, VMware vSphere, Tanzu, RedHat Openshift, Rubrik, Veeam, Commvault, Veritas, Cribl, and others.