Data Protection for Elasticsearch Environments with Cloudian HyperStore

Challenges

Data growth and the need to extract timely insights from this increasingly strategic asset are two of the biggest challenges for enterprises today. Elasticsearch, the leading open-source indexing, and search platform, is used by enterprises of all sizes to index, search, and analyze their data and gain valuable insights for making data-driven business decisions. Ensuring the durability of these valuable insights and accompanying data assets has become critical to enterprises for reasons ranging from compliance and archival to continued business success.

Today’s backup targets for Elasticsearch environments — tape, disk, and cloud — all have limitations that hinder your ability to meet your objectives.

  • Tape: Low-cost media, but concerns include reliability, long-term durability, and management workload.
  • Disk: A fast and effective solution but becomes costly as capacities grow.
  • Cloud: Low monthly cost, but often has access bandwidth issues that can limit the ability to meet SLAs. Additional costs associated with access and movement of data.

Resulting issues include excessive backup times — especially when backups fail — and the challenge of meeting RTO and RPO SLAs.

Solution

Cloudian HyperStore, the industry’s leading on-prem S3 based object storage platform, seamlessly integrates with Elasticsearch environments, as a target for API-driven data snapshot and restoration. The solution supports automated, policy-based snapshots of individual indices or of the entire cluster backed up to HyperStore using Elastic’s S3 plugin for snapshot and restore. Snapshots are taken incrementally avoiding the need to copy any data that is already stored in HyperStore as part of an earlier snapshot of the same index. Restoration of snapshots from HyperStore is allowed into an active cluster via the Elasticsearch restore API.

As an option, the solution also enables hybrid and multi-cloud snapshot and restore.

elasticsearch diagram
Figure 1: Cloudian HyperStore seamlessly integrates with Elasticsearch using the S3 Snapshot and Restore plugin as an intelligent, on-prem backup, archive, and DR storage target protecting one or all Elasticsearch indices.

Solution Advantages

Drop-in Integration
Cloudian HyperStore, the industry’s leading on-prem S3-based object storage platform, seamlessly integrates with Elasticsearch environments, as a target for API-driven data snapshot and restoration. The solution supports automated, policy-based snapshots of individual indices or backing up entire clusters to HyperStore using Elastic’s S3 plugin for snapshot and restore. Snapshots are taken incrementally, avoiding the need to copy any data that is already stored in HyperStore as part of an earlier snapshot of the same index. Restoration of snapshots is supported from HyperStore into an active cluster via the Elasticsearch restore API.

Performance to Handle the Largest Elasticsearch Environments
Cloudian scales to petabytes with a scaling model that grows in both capacity and bandwidth. Predictable backup windows result from Cloudian’s streaming bandwidth. Data write bandwidth in excess of 5,000MB/s (or 18TB per hour) can be achieved.

Petabyte-scalable
Unlike conventional storage, Cloudian offers modular growth, letting you expand from terabytes to an exabyte without disruption. Embedded data redundancy features provide up to 14 nines data durability, removing the necessity of a separate data backup process. Compared with traditional enterprise storage — or with storage on compute-intensive servers — Cloudian saves up to 70% on TCO. Cloudian saves on space, too, with the industry’s highest density: up to 1.5PB capacity in a 4U high chassis.

Secure
To ensure data security, Cloudian provides AES-256 server-side encryption for data at rest, SSL for data in transit (HTTPS), role-based access controls, storage policies that can be applied at an object and bucket-level management granularity.

Cloud Enabled
Cloudian is on-prem storage, but integrates directly with public cloud storage services. Employ policy-based tools to replicate or tier data to AWS, GCP, or Azure for offsite DR, capacity expansion, or data analysis in the cloud. This built-in capability requires no additional software or licenses.

Multi-Purpose Scalable Storage
Cloudian is the industry’s most compatible S3 API and integrates seamlessly with most S3-compatible applications. In addition to data protection for Elasticsearch environments, it can also provide scalable storage for data management applications from Rubrik, Veeam, Commvault, Veritas, Pure Storage, Quantum, and others.

Deploy as Software or Appliances
Cloudian is available as preconfigured appliances, with capacities from 96TB to 1.5PB, and as software for either bare-metal servers or VMs.

Solution Benefits

  • On-prem storage for business-critical snapshots of the entire Elasticsearch cluster, with exabyte scalability
  • Blazing fast restore and recovery from indexer node failures attributed to enterprise network vs restoring over WAN
  • Modular design for non-disruptive storage growth
  • Built-in data protection; up to 14 nines data durability
  • Public cloud compatible; integrates seamlessly with AWS, GCP, and Azure
  • Hybrid cloud-ready for DR
  • 70% TCO savings vs traditional storage and/or public cloud backups