What Is a Data Lake? Architecture and Deployment

Cloud vs. On-Premises Data Lakes

Most organizations traditionally deployed data lakes in their on-prem data centers. However, modern data lakes are often part of a cloud architecture today. Deploying data lakes in the cloud is more financially sound, especially with the new cloud-based services that support data lake integration, management, and automation.

The shift to the cloud followed the introduction of big data cloud platforms and various managed services incorporating technologies like Spark and Hadoop. Leading cloud providers like Google, Microsoft, and AWS now offer technology stacks for big data analytics projects, including Google Dataproc, Azure HDInsight, and Amazon EMR.

Another factor in the growing cloud data lake trends is the rise of cloud-based object storage, including services like Google Cloud Storage, Azure Blob Storage, and Amazon S3. These services offer a cost-effective alternative to traditional data storage options like Hadoop Distributed File System (HDFS).

Data Lake Architecture

Data lake architectures accommodate all data structures, including no structure, and support any format. Data lakes consist of two components: storage and compute. An entire data lake can reside on-premises or in a cloud environment. Some data lake architectures combine on-prem and cloud-based infrastructure.

Measuring the required capacity of a data lake is usually impossible because an organization’s data volumes change over time. Thus, the data lake architecture must be highly scalable. It should expand to accommodate petabytes or even exabytes of data. Traditional data storage solutions lack this capacity and flexibility.

Given the large amounts of data in a data lake, tagging objects with metadata is important to make them accessible in the future. The structure of a data lake’s software (i.e., S3, Hadoop) varies, but the objective is to make data easy to locate and use.

The data lake architecture should include the following features to ensure functionality and prevent it from turning into a data swamp:

• Data profiling—provides insights into object classifications and quality.
• Data classification taxonomy—describes use cases and content, data types, and user groups.
• Hierarchy—orders files and applies naming conventions.
• Access monitoring—tracks user access to the data lake and generates alerts specifying the time and location.
• Search functionality—allows users to find data.
• Data security—includes encryption, authentication, access control, and other mechanisms to prevent unwanted access.

Data Lake vs. Data Warehouse

Data lakes and data warehouses have similar basic objectives, but they are not interchangeable. Both storage systems consolidate data from different sources and provide a unified data store for multiple applications.

However, each storage repository suits different use cases due to the following key differences.

Schema

• Data warehouses have a schema-on-write model, meaning they require a defined, structured schema before storing data. Thus, most data preparation occurs before storage.
• Data lakes have a schema-on-read model, meaning they don’t require a predefined schema to store data. Processing occurs later when someone uses the data.

Accessibility

• Data warehouses offer relatively simple accessibility to users. Non-technical users and new team members can easily access data given the well-documented and clear schema.
• Data lakes are more complex because they don’t have organized data in a clear structure. They often require experts to understand the different data types to identify and read objects.

Flexibility

• Data warehouses are rigid and more time-consuming to set up and adjust. In addition to defining the schema before storing data, warehouses require significant resources to change the schema to meet new data requirements.
• Data lakes adapt more easily and don’t take much time to change. They are more scalable and can accommodate spikes in storage capacity demand.

Learn more in our detailed guide to data warehouse vs. data lake (coming soon)

What Is a Data Lakehouse?

A data lakehouse is a hybrid storage model that incorporates the advantages of data lakes and data warehouses. It balances a data lake’s large, unstructured mass, flexibility, and affordability and a data warehouse’s rigid structure and usability.

The concept of a data lakehouse introduces support for ACID processes:

• Atomicity
• Consistency
• Isolation
• Durability

These processes——allow multiple users to read and write data simultaneously. It should also have a mechanism for enforcing schemas and ensuring governance to maintain data integrity.

One reason for the emergence of data lakehouses is the growing presence of semi-structured and unstructured data in varied formats. This data is often useful for AI and machine learning (ML) applications and includes images, video, audio, and text.

A data lakehouse thus supports many different workloads. For example, data warehouses often involve databases, while data lakes enable ML, data science, SQL, and other analytics workflows.

Another major benefit of a data lakehouse is that it lets users easily access a wider range of data types using different tools. It integrates with enterprise apps and speeds up many data-driven tasks.

Learn more in our detailed guide to data lakehouse (coming soon)

How to Implement a Data Lake Solution

The process of implementing a data lake should include the following key steps:

• Determine the skill level—identify the expertise required to support the storage platform and perform data analytics tasks. A data lake is a complex technology with a steep learning curve, requiring organizations to employ experienced personnel or train existing employees. Organizations must define new roles and processes for reporting.
• Establish objectives—create a clear data lake design and implementation strategy, specifying goals, processes, and milestones. Identify the organization’s criteria for evaluating the data lake’s success and design the storage system to enable data analysis. Establish standards for classifying and storing data.
• Prioritize data—evaluate the data sources to determine their importance to the organization. The data lake can ingest any data generated by the organization, so prioritization is key.
• Evaluate data analysis—determine what information requires analysis and at what level. More thoroughly analyzed data may be a lower priority.
• Implement data governance—establish a governance strategy and apply measures to enforce it and ensure the security and integrity of data.
• Identify use cases for the data—set standards for experimentation, exploration, and analysis. Provide a flexible, standardized process for data scientists to evaluate data and determine how it can generate business value. Other applications and business intelligence (BI) platforms are potential targets.

Related content: Read our guide to S3 data lake (coming soon)

Data Protection with Cloudian

Data protection requires powerful storage technology. Cloudian’s storage appliances are easy to deploy and use, let you store Petabyte-scale data and access it instantly. Cloudian supports high-speed backup and restore with parallel data transfer (18TB per hour writes with 16 nodes).

Cloudian provides durability and availability for your data. HyperStore can backup and archive your data, providing you with highly available versions to restore in times of need.

In HyperStore, storage occurs behind the firewall, you can configure geo boundaries for data access, and define policies for data sync between user devices. HyperStore gives you the power of cloud-based file sharing in an on-premise device, and the control to protect your data in any cloud environment.

Learn more about data protection with Cloudian.