Building and Protecting Data Lakehouse Projects with Cloudian and Vertica

Over the past year, Cloudian has greatly expanded its support for data analytics through new partnerships. One of those key partnerships is with Vertica, where the combination of Vertica and Cloudian HyperStore enables organizations to build and protect data lakehouses for modern data analytics applications.

This blog highlights the three main use cases we’re currently serving together:

• Starting a data lakehouse with Vertica in Eon mode and Cloudian
• Extending the data lakehouse with Vertica external tables and Cloudian
• Protecting Vertica datasets with data backup to Cloudian

Just as a reminder, Vertica is a unified analytics data warehouse platform, based on a massively scalable architecture, and Cloudian is a software-defined, limitlessly scalable, S3-compatible object storage platform for on-premises and hybrid cloud environments.

Starting a Data Lakehouse with Vertica in Eon Mode and Cloudian

In the data analytics space, Vertica is known for performance, whether it is run in “Enterprise Mode” or “Eon Mode.” In Enterprise Mode each database node stores a portion of the dataset and performs a portion of the computation. In Eon Mode, Vertica brings its cloud architecture to on-premises deployments and decouples compute and storage. In Eon Mode, each Vertica node can access a shared communal storage space via S3 API. The advantages are: a) compute can be scaled as required without having to scale storage, meaning no more server sprawl and b) storage can be consolidated into a single platform and accessed by various data tools:

Building out Vertica communal storage on Cloudian is easy. For this exercise we are going to assume we have both a functional Vertica and Cloudian HyperStore instance that can communicate via HTTP(s):

1. Configure a bucket via Cloudian Management Console (CMC) on your HyperStore cluster:
   • • Let’s use the name “verticabucketoncloudian” for this example.

   

2. Create an auth_params.conf file:
   • On your Vertica node, create an auth_params.conf file that will be accessible when you create the Vertica database instance.
     auth_params.conf values required are going to be:
     
awsauth = Access_Key:Secret_Key
awsendpoint = HyperstoreAddress:Port (either 443 or 80)
awsenablehttps = 0 Is required if not using HTTPs

3. Create your Vertica in Eon Mode database instance:
   • On your Vertica node, create the database instance. Specify the location of your auth_params.conf file to leverage a Cloudian S3 bucket for communal storage.
     
admintools -t create_db -x auth_params.conf \
--communal-storage-location=s3://verticabucketoncloudian \
--depot-path=/home/dbadmin/depot --shard-count=6 \
-s vnode01,vnode02,vnode03,vnode04,vnode05,vnode06 -d verticadb -p 'YourDBAdminPasswordHere'

4. Success! Let’s test.
   • Once the above command returns successfully, you can test the Vertica in Eon Mode instance.
   • Connect to your db instance and load a dataset.
   • Connect to Cloudian bucket “verticabucketoncloudian” via CMC or S3 browser, and you will see objects in the bucket.

Extending the Data Lakehouse with Vertica External Tables and Cloudian

One of the key tenants of a successful data lakehouse initiative is the ability to access and analyze datasets that have been generated by other analytics platforms.

Prior to the data lakehouse, an ETL (Extract Transform Load) operation would have been required to move data from one analytics platform to another. Today, Vertica can analyze the data in-place by leveraging external tables, without the need for complex and expensive data moves.