Cloudian recently introduced a new capability, hybrid storage policies, first introduced in Cloudian HyperStore 7.5 object storage software. This feature allows the Cloudian platform to be application aware and tune storage usage in real time to optimize performance and capacity efficiency. A customer recently capitalized on this new capability as they sought to bolster their global file sharing capabilities with Nasuni’s global file system, a platform that provides data access across distributed locations while maintaining data consistency.
Challenge: How to Maximize Performance on Object Storage
For their Nasuni storage repository, the customer selected Cloudian HyperStore object storage software. The initial test deployment with Nasuni and Cloudian proceeded without a hitch. Cloudian’s native S3 API, renowned for its comprehensive S3 compatibility, seamlessly integrated with Nasuni. The performance, however, fell short of the firm’s aspirations for their global deployment. The firm’s IT team recognized the need for enhanced file performance and engaged Cloudian for a solution.
An analysis of the workload profile showed that the Nasuni platform drove a significant volume of small object transfers to and from the storage platform. This phenomenon stemmed from the creation of hundreds of millions of small objects, each around 4K bytes in size, within a 100TB data capacity.
The root of the performance issue traced back to the system configuration. The Cloudian system had been configured to optimize storage efficiency with erasure coding, a scheme that protects data by striping it across multiple Cloudian nodes. While this configuration is optimal for larger objects, for small objects (as in this Nasuni use case) it incurred processing overhead that impeded performance.
One potential solution was to change the data protection scheme: replicate data across nodes instead of striping it. While this approach would alleviate processing overhead, it would also compromise storage efficiency for larger objects. Given the multi-faceted nature of the Cloudian system’s workloads, this global configuration change was deemed impractical.
Solution: The Hybrid Storage Policy
To address this challenge, the Cloudian team proposed a hybrid storage policy. This innovative feature empowered the storage system to discern the workload and dynamically adjust the storage policy to achieve optimal outcomes. For smaller objects, the system would employ data replication to minimize latency. Given the tiny size of the objects generated by the Nasuni application, the reduced storage efficiency would have minimal impact on the storage capacity consumed. For larger objects, the hybrid policy would automatically use a data striping policy to maximize storage efficiency. This would achieve the best of both worlds: high storage efficiency where it is required, and high random throughput where it is needed.
The Advantages of a Hybrid Policy
The hybrid storage policy emerged as a valuable add, optimizing both performance and efficiency. Cloudian HyperStore offered the flexibility of replicating data across various devices or striping it across multiple devices to safeguard against device or site failures. Each approach had its merits—striping economized space for larger objects, while replication boosted performance for smaller ones. The hybrid policy offered the best of both realms, seamlessly adapting to the storage workload without necessitating additional management efforts.
Learn more about Cloudian HyperStore here.
