Global Retailer Streamlines Splunk Data Analytics with a Cloudian AI Data Lake

Jon Toor

storage for splunk data A prominent global retailer faced challenges in managing the voluminous data generated by its Splunk analytics platform. To address these challenges, the company embarked on a strategic initiative to modernize its data storage infrastructure by integrating Cloudian’s S3-compatible AI data lake. This case study outlines the retailer’s journey from relying solely on HPE 3PAR/Primera Block Storage for Splunk data to implementing Cloudian, resulting in significant cost savings, enhanced data sovereignty, and improved data availability across its operations.

Challenge

The retailer’s Splunk infrastructure, a crucial component for data analytics and insights, was primarily based on the HPE Synergy compute platform, supported by HPE 3PAR/Primera block-based storage. With the Splunk application being one of the largest storage consumers at the firm, the backend storage footprint had escalated to nearly 2 petabytes (PB). This significant demand for storage not only resulted in high costs but also posed scalability and data sovereignty concerns.

Selection of Cloudian

The firm identified the need for a modern storage solution that could seamlessly integrate with Splunk, particularly one that supports the S3 protocol for enabling Splunk SmartStore capabilities. Cloudian emerged as the preferred choice due to its S3 compatibility, cost-effectiveness, and certification for use with SmartStore.

Implementation Highlights

SmartStore Enablement: By leveraging Splunk’s native support for S3-compatible storage targets, the retailer was able to offload older, less frequently accessed data (ie, “warm” and “cold” data) from expensive block storage to Cloudian, reserving block storage primarily for “hot” data.
On-Prem Data Sovereignty: In keeping with the firm’s data sovereignty requirements, Cloudian’s solution was implemented on-premises, ensuring that Splunk data remained within the firm’s control.
Distributed Erasure Coding: The deployment featured a robust erasure coding configuration that distributed data across three data centers. This redundancy ensured high data availability and resilience, capable of withstanding the loss of an entire data center while maintaining full data read/write capabilities.

Outcome

Significant Cost Reduction: The switch to Cloudian produced a 51% reduction in costs associated with Splunk data storage by offloading warm and cold data to a more economical storage solution. Moreover, the Splunk team reported satisfaction with Cloudian’s performance, especially when fetching older data.
Enhanced Data Availability: Cloudian demonstrated the highest data availability of all storage types used by the firm, including block, file, hyper-converged infrastructure, and cloud storage solutions.
Flexible and Scalable Storage: Cloudian’s multi-purpose and S3-compatible storage solution not only met the immediate needs for Splunk data storage but also positioned the firm to efficiently manage unstructured data for various use cases in the future.

Conclusion

The global retailer’s strategic deployment of Cloudian for Splunk data analytics represents a significant leap forward in managing large-scale data with enhanced efficiency, security, and cost-effectiveness. By adopting Cloudian’s modern data lake solution, the firm not only addressed its immediate storage challenges but also laid a solid foundation for future data management and analytics initiatives. This case study demonstrates the transformative potential of integrating advanced storage solutions with data analytics platforms, driving operational excellence and strategic insights in a competitive retail landscape.