S3 Buckets: Accessing, Managing, and Securing Your Buckets

Amazon Simple Storage Service (Amazon S3) is an object storage solution that provides data availability, performance, security and scalability. Organizations from all industries and of every size may use Amazon S3 storage to safeguard and store any amount of information for a variety of use cases, including websites, data lakes, backup and restore, mobile applications, archives, big data analytics, IoT devices, and enterprise applications.

What Is AWS S3 Bucket?

Amazon Simple Storage Service (Amazon S3) is an object storage solution that provides data availability, performance, security and scalability. Organizations from all industries and of every size may use Amazon S3 storage to safeguard and store any amount of information for a variety of use cases, including websites, data lakes, backup and restore, mobile applications, archives, big data analytics, IoT devices, and enterprise applications.

To retain your information in Amazon S3, you use resources called objects and buckets. A bucket is a container that houses objects. An object contains a file and all metadata used to describe the file.

To retain an object in Amazon S3, you develop a bucket and upload the object into it. Once the object is within the bucket, you may move it, download it, or open it. When you don’t require the bucket or object any longer, you can discard them to trim back on your resources.

In this article:

 

This is part of an extensive series of articles about S3 Storage.

How to Use an Amazon S3 Bucket

An S3 customer starts by establishing a bucket in the AWS region of their choosing and assigns it a unique name. AWS suggests that customers select regions that are geographically close to them in order to minimize costs and latency.

After creating the bucket, the user chooses a storage tier based on the usage requirements for the data—there are various S3 tiers ranging in terms of price, accessibility and redundancy. A single bucket can retain objects from distinct S3 storage tiers.

The user may then assign particular access privileges regarding the objects retained in the bucket using various mechanisms, including bucket policies, the AWS IAM service, and ACL.

An AWS customer may work with an Amazon S3 bucket via the APIs, the AWS CLI, or the AWS Management Console.

Related content: Read our guide to the S3 API

Tutorial: Creating a Bucket

Before you can store content in S3, you need to open a new bucket, selecting a bucket name and Region. You may also wish to select additional storage management choices for your bucket. Once you have configured a bucket, you can’t modify the Region or bucket name.

The AWS account that opened the bucket remains the owner. You may upload as many objects as you like to the bucket. According to the default settings, you can have as many as 100 buckets for each AWS account.

S3 lets you create buckets using the S3 Console or the API.

Keep in mind that buckets are priced according to data volume stored in them, and other criteria. Learn more in our guide to S3 pricing

Developing an S3 bucket via the S3 console:

  1. Access the S3 console.
  2. Select Create bucket.
  3. In Bucket name, create a DNS-accepted name for your bucket.

 

Image Source: AWS

The bucket name must be unique, begin with a number or lowercase letter, be between 3-63 characters, and may not feature any uppercase characters.

4. Select the AWS Region for the bucket. Select a Region near you to keep latency and cost to a minimum and to address regulatory demands. Keep in mind there are special charges for moving objects outside a region.
5. In Bucket settings for Block Public Access, specify if you want to allow or block access from external networks.
6. You can optionally enable the Object Lock feature in Advanced settings > Object Lock.
7. Select Create bucket.

What Is S3 Bucket Policy?

S3 provides the concept of a bucket policy, which lets you define access permissions for a bucket and the content stored in it. Technically, it is an Amazon IAM policy, which employs a JSON-based policy language.

For instance, policies permit you to:

  • Enable read access for unknown users
  • Restrict a particular IP address from accessing the bucket
  • Place a limit on access to a particular HTTP referrer
  • Require multi-factor authorization

S3 Bucket URLs and Other Methods to Access Your Buckets

You can perform almost any operation using the S3 console, with no need for code. However, S3 also provides a powerful REST API that gives you programmatic access to buckets and objects. You can reference any bucket or the objects within it via a unique Uniform Resource Identifier (URI).

Amazon S3 provides support for path-style and virtual-hosted-style URLs to gain access to a bucket. Given that buckets are accessible to these URLs, it is suggested that you establish buckets with bucket names that are DNS-compliant.

Virtual-Hosted-Style Access

In a virtual-hosted-style request, the bucket name is a component of the domain name within the URL.

Amazon S3 virtual-hosted-style URLs employ this format:

https://bucket-name.s3.Region.amazonaws.com/key name

For example, if you name the bucket bucket-one, select the US East 1 (Northern Virginia) Region, and use kitty.png as your key name, the URL will look as follows:

https://bucket-one.s3.us-east-1.amazonaws.com/kitty.png

Path-Style Access

In Amazon S3, path-style URLs use this format:

https://s3.Region.amazonaws.com/bucket-name/key name

For example, if you created a bucket in the US East (Northern Virginia) Region and named it bucket-one, the path-style URL you use to access the kitty.jpg object in the bucket will look like this:

https://s3.us-east-1.amazonaws.com/bucket-one/kitty.jpg

Accessing a Bucket Via S3 Access Points

As well as working with a bucket directly, you can work with a bucket via an access point.

S3 access points exclusively support virtual-host-style addressing. To address a bucket via an access point, you must employ the following format:

https://AccessPointName-AccountId.s3-accesspoint.region.amazonaws.com.

Accessing a Bucket Using S3://

Certain AWS services need you to specify an Amazon S3 bucket via S3://bucket, where you will need to follow this format:

S3://bucket-name/key-name

Note that when employing this format the bucket name does not feature the AWS Region. For example, a bucket called bucket-one with a kitty.jpg key will look like this:

S3://bucket-one/kitty.jpg

S3 Bucket Configuration: Understanding Subresources

AWS provides various tools for Amazon S3 buckets. An IT specialist may enable different versions for S3 buckets to retain every version of an object when an operation is carried out on it, for example a delete or copy operation. This may help stop IT specialists from accidentally deleting an object. Similarly, when creating a bucket, a user can establish server access logs, tags, object-level API logs, and encryption.

S3 Transfer Acceleration can assist with the execution of secure and fast transfers from the client to an S3 bucket via AWS edge locations.

Amazon S3 provides support for different alternatives for you to configure your bucket. Amazon S3 offers support for subresources so you can manage and retain the bucket configuration details. You can employ the Amazon S3 API to manage and develop these subresources. You may also utilize the AWS SDKs or the console.

These are known as subresources since they function in the context of a certain object or bucket. Below lists subresources that let you oversee bucket-specific configurations.

cors (cross-origin resource sharing):  You may configure your bucket to permit cross-origin requests.

event notification: You may permit your bucket to alert you of particular bucket events.

lifecycle: You may specify lifecycle regulations for objects within your bucket that feature a well-outlined lifecycle.

location: When you establish a bucket, you choose the AWS Region where you want Amazon S3 to develop the bucket. Amazon S3 retains these details in the location subresources and offers an API so you can gain access to this information.

logging: Logging lets you monitor requests for access to the bucket. All access log records give details regarding one access request, including bucket name, requester, request action, request time, error code, and response status.

object locking: Enables the object lock feature for a bucket. You may also wish to configure a default period of retention and mode that applies to the latest objects that are uploaded to the bucket.

policy and ACL (access control list): Both buckets and the objects stored within them are private, unless you specify otherwise. ACL and bucket policies are two ways to grant permissions for an entire bucket.

replication: This option lets you automatically copy the content of the bucket to additional buckets, within the Amazon Region. Replication is asynchronous.

requestPayment: By default, the AWS account that sets up a bucket also receives bills for requests made to the bucket. This setting lets the bucket creator pass on the cost of downloading data from the bucket to the account downloading the content.

tagging: This setting allows you to add tags to an S3 bucket. This can help you track and organize your costs on S3. AWS shows the tags on your charges allocation report, with costs and usage aggregated via the tags.

transfer acceleration: Transfer acceleration enables easy, secure and fast movement of files over extended distances between your S3 bucket and your client. Transfer acceleration leverages the globally distributed edge locations via Amazon CloudFront.

versioning: Versioning assists you when recovering accidental deletes and overwrites.

website: You may configure the bucket for static website hosting.

Best Practices for Keeping Amazon S3 Buckets Secure

AWS S3 Buckets may not be as safe as most users believe. In many cases, AWS permissions are not correctly configured and can expose an organization’s AWS S3 buckets or some of their content.

Although misconfigured permissions are by no means a novel occurrence for many organizations, there is a specific permission that entails increased risk. If you allow objects to be public, this establishes a pathway for cyberattackers to write to S3 buckets that they don’t have the correct permissions to access. Misconfigured buckets are a major root cause behind many well-known attacks.

To protect your S3 buckets, you should apply the following best practices.

Block Public S3 Buckets at the Organization Level

Assign AWS accounts for public S3 utilization and stop all other S3 buckets from accidentally becoming public by putting in place S3 Block Public Access. Employ Organizations Service control policies (SCPs) to ensure that the Block Public Access setting is not alterable. S3 Block Public Access offers a degree of safety that functions at the level of the account and also on single buckets, encompassing those that you develop in the future.

You retain the capacity to prevent existing public access—irrespective of whether it was specified by a policy or an ACL—and to make sure that public access is not given to items you newly create. This provides only specific AWS accounts with public S3 buckets and stops all other AWS accounts.

Implement Role-Based Access Control

Outline roles that cover the access needs of users and objects. Make sure those roles have the least access needed to carry out the job so that if a user’s account is breached, the damage is kept to a minimum.

AWS security is founded on AWS Identity and Access Management (IAM) strategies. A principal is an identity that may be validated, for example, with a password. Roles, users, applications, and federated users (from separate systems) may all be principals. When a validated principal requests an entity, resource, service, or a different asset, verification begins.

Verification policies determine what access the principal has to the resource being requested. Approval is given based on resource-based methods or identity. Matching each validated principal with each validated policy will ascertain if the request is permitted.

Another data security methodology is splitting or sharing data into different buckets. For instance, a multi-tenant application could require separate Amazon S3 buckets for every tenant. You can use another AWS tool, Amazon VPC, which grants your endpoints secure access to sections of your Amazon S3 buckets.

Encrypt Your Data

Even with your greatest efforts, it remains good practice to assume that information is always at risk of being exposed. Given this, you should use encryption to stop unauthorized individuals from using your information if they have managed to access it.

Make sure that your Amazon S3 buckets are encrypted during transit and while sitting on the server. If you just have a single bucket, this is likely not complex, but if buckets are being developed dynamically, it may be difficult to keep track of them and manage encryption appropriately.

On the server side, Amazon S3 buckets support encryption, but this has to be enabled. Once encryption is turned on, the information is encrypted at rest. Encrypting the bucket will make sure that any individual who manages to access the data will require a password (key) to decrypt the data.

For transport security, HTTPS is used to make sure that information is encrypted from one end to another. Every additional version of Transport Layer Security (TLS) ensures that the protocol is more secure and does away with out-of-date, now insecure, encryption methods.

S3-Compatible Storage On-Premises with Cloudian

Cloudian® HyperStore® is a massive-capacity object storage device that is fully compatible with Amazon S3. It allows you to easily set up an object storage solution in your on-premises data center, enjoying the benefits of cloud-based object storage at much lower cost.

HyperStore can store up to 1.5 Petabytes in a 4U Chassis device, allowing you to store up to 18 Petabytes in a single data center rack. HyperStore comes with fully redundant power and cooling, and performance features including 1.92TB SSD drives for metadata, and 10Gb Ethernet ports for fast data transfer.

cloudian hyperstore appliance

HyperStore is an object storage solution you can plug in and start using with no complex deployment. It also offers advanced data protection features, supporting use cases like compliance, healthcare data storage, disaster recovery, ransomware protection and data lifecycle management.

Learn more about Cloudian® HyperStore®.

Object Storage in the Cloud: 4 Providers Compared

What Is Object Storage in the Cloud?

As your business expands, you have to manage isolated but rapidly growing pools of data from various sources, which are used for a variety of business processes and applications. Nowadays, many organizations grapple with a fragmented storage portfolio that slows down innovation and adds complexity to an organization’s applications. Object storage can help your organization break down these silos. It provides cost-effective, highly scalable storage that can retain any type of data in its original format.

Object storage is highly suitable for the cloud as it is flexible, elastic and can be more easily scaled into many petabytes to support indefinite data growth. The architecture manages and stores data as objects, as opposed to block storage, which relates to data as logical volumes, blocks and files storage, where data is stored in hierarchical files.

Related content: Read our guides on object storage vs block storage and object storage vs file storage.

In this article:

4 Cloud Object Storage Options

Let’s review the object storage offerings by some of the world’s leading cloud providers: Amazon Web Services, Microsoft Azure, Google Cloud, and IBM Cloud.

AWS Object Storage

AWS provides a distinct variety of storage classes for different use cases. Amazon S3 is the main object storage platform of AWS, with S3 Standard-IA providing cool storage, and Glacier providing cold storage:

  • Amazon S3 Standard—this is the storage choice for information that is often accessed, and is great for numerous use cases including dynamic websites, cloud applications, content distribution, data analytics and gaming. It delivers high throughput as well as low latency.
  • Amazon S3 Standard-Infrequent Access (Amazon S3 Standard—IA)—this is a storage alternative for data which is accessed less often, such as disaster recovery and long-term backups.
  • Amazon Glacier—this highly durable storage system is optimized for data that is not often accessed, or “cold” data, such as end-of-lifecycle data kept for compliance and regulatory backup purposes. Data is archived for long-term storage, and is immutable and encrypted.

Azure Object Storage

Microsoft offers Azure Blob Storage for object storage in the cloud. Blob storage is suited to storing any form of unstructured data, such as binary or text. This includes videos, images, documents, audio and more. Azure storage offers high-quality data integrity, flexibility and mutability.

Blob storage is employed for serving documents or images directly to a browser, for retaining files for distributed access, streaming audio and video, writing to log files, disaster recovery, storing data for restore and backup, and archiving, so it can be analyzed by an Azure-hosted or on-premises service.

Azure has several storage tiers, including:

  • Hot access tier— for information that is in or anticipated to be in active use and staged for processing and subsequent migration to the Cool storage tier.
  • Cool access tier—for data that is intended to stay in the Cool tier for more than 30 days. This includes disaster recovery datasets and short-term backup, media content that is older and intended to be immediately available when drawn on and large data sets.
  • Archive access tier—for data which will stay in the Archive tier for more than 180 days, and which can tolerate hours of retrieval latency.

Note: The Archive storage tier is not accessible at the storage account level, but only at the blob level. Azure also provides a Premium tier, which is for workloads that need consistent and fast response times.

Google Cloud Storage

Google Cloud Storage (GCS) provides united object storage for all workloads. It has four classes for backup and archival storage and high-performance object storage. All four classes provide high durability and low latency:

  • Hot (high-performance) storage—GCS provides regional and multi-regional storage for high-frequency access information.
  • Multi-regional storage—allows for the storing of information that is often accessed around the world, including streaming videos, serving website content, or mobile and gaming applications.
  • Regional storage—allows for frequent access to information in the corresponding region of Google Compute Engine instance or Google Cloud DataProc, for example data analytics.
  • Nearline (cool) storage—for data that only needs to be accessed less than once a month, but several times a year. Suitable for backups and long-tail multimedia content.
  • Coldline (cool) storage—for data that only needs to be accessed less than once a year. Suitable for archival data and disaster recovery.

IBM Cloud Object Storage

IBM Cloud provides scalable and flexible cloud storage with policy-driven archive abilities for unstructured data. This cloud storage service is intended for data archiving, for example for the long term retention of data that is infrequently accessed, including for mobile and web applications, and for backup and analytics.

IBM has four storage-class tiers integrated with an Asperaâ high-speed information transfer option. This allows for the easy transfer of data from and to Cloud Object Storage, and query-in-place functionality.

IBM Cloud Object Storage class tiers:

  • Standard storage—for active workloads that need high performance and low latency, and data that requires frequent and multiple access in a month. Usage scenarios are for example, active content repositories, analytics, mobile streaming and web content, collaboration and DevOps.
  • Vault storage—for less active workloads which need real-time, on-demand access but only infrequently, up to once a month. Use cases include digital asset retention and backup.
  • Cold vault—for cold workloads, where data needs on-demand, real-time access when needed but is mainly archived. For example, data that is accessed several times a year. Common use cases involve long-term backup, large data set preservation such as older media content and scientific data.
  • Flex storage—this class tier is utilized for dynamic workloads (combining cold and hot workloads) and data based on access patterns. Typical use cases include cognitive workloads, cloud-native analytics and user-generation applications.

Cloud Object Storage Pros and Cons

The following are some of the key advantages and disadvantages of object storage in the cloud.

Cloud Object Storage Pros

The key advantages of object storage include:

  • Data is highly distributed, which ensures it is more resilient to hardware failures or disasters. This way, it is available even if various nodes fail.
  • Objects are kept in a flat address space, which minimizes complexity and scalability issues.
  • Data protection is built into this architecture in the form of erasure coding or replication technology.
  • Object storage is most suitable for cloud storage and static data. Common use cases for object storage include archiving and cloud backup—the technology functions best with data that is more frequently read than written to.
  • Object storage has developed to the point where it scales at the exabyte level and represents trillions of objects. The use of VMs or commodity hardware enables nodes to be added easily, with the disk space being used more efficiently.
  • Object storage systems, via the use of object IDs (OIDs) or identifiers, can gain access to any piece of data without knowing on which physical storage device, directory, or file system it resides on. The abstraction lets object storage devices operate with storage hardware configured in distributed node architecture. This way, processing power can scale together with data storage capacity.
  • I/O requests don’t need to pass via a central controller, allowing for a true global storage system for large amounts of data overseen by objects, physically kept anywhere, and retrieved through the internet or a WAN.

Cloud Object Storage Pros

The key disadvantages of object storage include:

  • Object storage systems are not steady enough for real-time systems, including transactional databases. An undesirable use case for object storage is an environment or application with a high transactional rate.
  • Object storage doesn’t guarantee that read requests will produce the most up-to-date version of the data.
  • This technology isn’t alway appropriate for applications that have high performance demands.
  • Cloud-based storage often ends up being more expensive because you need to pay for storage on an ongoing basis. With on-premises equipment you pay once and the storage is yours.

Bring Object Storage On-Premises with Cloudian

Cloudian® HyperStore® is a massive-capacity object storage device that is fully compatible with Amazon S3. It allows you to easily set up an object storage solution in your on-premises data center, enjoying the benefits of cloud-based object storage at much lower cost.

HyperStore can store up to 1.5 Petabytes in a 4U Chassis device, allowing you to store up to 18 Petabytes in a single data center rack. HyperStore comes with fully redundant power and cooling, and performance features including 1.92TB SSD drives for metadata, and 10Gb Ethernet ports for fast data transfer.

HyperStore is an object storage solution you can plug in and start using with no complex deployment. It also offers advanced data protection features, supporting use cases like compliance, healthcare data storage, disaster recovery, ransomware protection and data lifecycle management.

Learn more about Cloudian® HyperStore®.

What Is Object Storage: Definition, How It Works, and Use Cases

What Is Object Storage?

Object storage is a data storage architecture that stores and manages unstructured data in units called objects. Objects can be any size or format, and can include data, metadata, and a unique identifier.

Unlike other storage systems, object storage is not organized into folders or a hierarchical path, so objects can be reached through multiple paths. In object storage, objects are stored in a flat data environment and can be accessed through multiple paths, rather than being organized into folders.

Objects can store photos, videos, emails, audio files, network logs, or any other type of structured or unstructured data. All of the major public cloud services, including Amazon, Google and Microsoft, employ object storage as their primary storage.

This is part of an extensive series of guides about data security.

In this article:

Object Storage Definition

Object storage is a technology that manages data as objects. All data is stored in one large repository which may be distributed across multiple physical storage devices, instead of being divided into files or folders.

It is easier to understand object-based storage when you compare it to more traditional forms of storage – file and block storage.

storage types including object storage

File Storage

File storage stores data in folders. This method, also known as hierarchical storage, simulates how paper documents are stored. When data needs to be accessed, a computer system must look for it using its path in the folder structure.

File storage uses TCP/IP as its transport, and devices typically use the NFS protocol in Linux and SMB in Windows.

Block Storage

Block storage splits a file into separate data blocks, and stores each of these blocks as a separate data unit. Each block has an address, and so the storage system can find data without needing a path to a folder. This also allows data to be split into smaller pieces and stored in a distributed manner. Whenever a file is accessed, the storage system software assembles the file from the required blocks.

Block storage uses FC or iSCSI for transport, and devices operate as direct attached storage or via a storage area network (SAN).

Object Storage

In object storage systems, data blocks that make up a file or “object”, together with its metadata, are all kept together. Extra metadata is added to each object, which makes it possible to access data with no hierarchy. All objects are placed in a unified address space. In order to find an object, users provide a unique ID.

Object-based storage uses TCP/IP as its transport, and devices communicate using HTTP and REST APIs.

Metadata is an important part of object storage technology. Metadata is determined by the user, and allows flexible analysis and retrieval of the data in a storage pool, based on its function and characteristics.

The main advantage of object storage is that you can group devices into large storage pools, and distribute those pools across multiple locations. This not only allows unlimited scale, but also improves resilience and high availability of the data.

Object Storage Architecture: How Does It Work?

Anatomy of an Object

Object storage is fundamentally different from traditional file and block storage in the way it handles data. In an object storage system, each piece of data is stored as an object, which can include data, metadata, and a unique identifier, known as an object ID. This ID allows the system to locate and retrieve the object without relying on hierarchical file structures or block mappings, enabling faster and more efficient data access.

Objects can be any size or format, and can store photos, videos, emails, audio files, network logs, or any other type of structured or unstructured data.

Data Storage Layer: Flat Data Environment

The data storage layer is where the actual data objects are stored. Object storage is not organized into folders or a hierarchical path, so objects can be reached through multiple paths. Objects are stored in a flat data environment and can be accessed through multiple paths.

In an object storage system, data is typically distributed across multiple storage nodes to ensure high performance, durability, and redundancy. Each storage node typically contains a combination of hard disk drives (HDDs) and solid-state drives (SSDs) to provide the optimal balance between capacity, performance, and cost. Data objects are automatically replicated across multiple nodes, ensuring that data remains available and protected even in the event of hardware failures or other disruptions.

Metadata Index

The metadata index is a critical component of object storage architecture, as it maintains a record of each object’s unique identifier, along with other relevant metadata, such as access controls, creation date, and size. This information is stored separately from the actual data, allowing the system to quickly and efficiently locate and retrieve objects based on their metadata attributes. The metadata index is designed to be highly scalable, enabling it to support millions or even billions of objects within a single object storage system.

API Layer

The API layer is responsible for providing access to the object storage system, allowing users and applications to store, retrieve, and manage data objects. Most object storage systems support a variety of standardized APIs, such as the Simple Storage Service (S3) API from Amazon Web Services (AWS), the OpenStack Swift API, and the Cloud Data Management Interface (CDMI). These APIs enable developers to easily integrate object storage into their applications, regardless of the underlying storage technology or vendor.

5 Expert Tips to help you better optimize your object storage care

Jon Toor, CMO

With over 20 years of storage industry experience in a variety of companies including Xsigo Systems and OnStor, and with an MBA in Mechanical Engineering, Jon Toor is an expert and innovator in the ever growing storage space.

Leverage lifecycle policies to manage storage costs: Implement object lifecycle management to automatically transition objects between storage classes based on their age or access patterns. This can help you reduce storage costs by moving infrequently accessed data to colder storage tiers.

Optimize metadata for faster search and analytics: Invest time in designing your object metadata schema. Adding meaningful, searchable metadata can dramatically enhance retrieval speed and enable powerful analytics without needing to process the entire object.

Use erasure coding for efficient data protection: While replication is common, erasure coding provides more efficient storage utilization, especially in environments with large datasets. It offers high durability while using less storage space than simple replication.

Enable versioning for data integrity and compliance: Activate object versioning to protect against accidental overwrites or deletions. This is critical for compliance in industries where data integrity is required over long retention periods.

Implement policy-driven data tiering: Automate data movement between hot, warm, and cold storage using policy-based rules. This approach allows you to maximize cost efficiency by aligning storage costs with data value and access frequency.

Object Storage Benefits

Exabyte Scalable

Unlike file or block storage, object storage services enable scalability that goes beyond  exabytes. While file storage can hold many millions of files, you will eventually hit a ceiling. With unstructured data growing at 50+% per year, more and more users are hitting those limits, or they expect to in the future.

Scale Out Architecture

Object storage makes it easy to start small and grow. In enterprise storage, a simple scaling model is golden. And scale-out storage is about as simple as it gets: you simply add another node to the cluster and that capacity gets folded into the available pool.

HyperStore is an S3-compatible storage system. HyperFile is a connector that allows files to be stored on HyperStore.

Customizable Metadata

While file systems have metadata, the information is limited and basic (date/time created, date/time updated, owner, etc.). Object storage allows users to customize and add as many metadata tags as they need to easily locate the object later. For example, an X-ray could have information about the patient’s age and height, the type of injury, etc.

High Sequential Throughput Performance

Early object storage systems did not prioritize performance, but that’s now changed. Now, object stores can provide high sequential throughput performance, which makes them great for streaming large files. Also, object storage services help eliminate networking limitations. Files can be streamed in parallel over multiple pipes, boosting usable bandwidth.

Flexible Data Protection Options

To safeguard against data loss, most traditional storage options utilize fixed RAID groups (groups of hard drives joined together), sometimes in combination with data replication. The problem is, these solutions generally lead to one-size-fits-all data protection. You can not vary the protection level to suit different data types.

Object storage solutions employ a flexible tool called erasure coding that is similar to old-fashioned RAID in some ways, but is far more flexible. Data is striped across multiple drives or nodes as needed to achieve the needed protection for that data type. Between erasure coding and configurable replication, data protection is both more robust and more efficient.

Support for the S3 API

Back when object storage solutions were launched, the interfaces were proprietary. Few application developers wrote to these interfaces. Then Amazon created the Simple Storage Service, or “S3”. They also created a new interface, called the “S3 API”. The S3 API interface has since become a de-facto standard for object storage data transfer.

The existence of a de facto standard changed the game. Now, S3-compatible application developers have a stable and growing market for their applications. And service providers and S3-compatible storage vendors such as Cloudian have a growing user set deploying those applications. The combination sets the stage for rapid market growth.

Lower Total Cost of Ownership (TCO)

Cost is always a factor in storage. And object storage services offer the most compelling story, both in hardware/software costs and in management expenses. By allowing you to start small and scale, this technology minimizes waste, both in the form of extra headcount and unused space. Additionally object storage systems are inherently easy to manage. With limitless capacity within a single namespace, configurable data protection, geo replication, and policy-based tiering to the cloud, it’s a powerful tool for large-scale data management.

To learn more about Cloudian’s fully native S3-compatible storage in your data center, and how it can cut down your TCO, check out our free trial. Or visit cloudian.com for more information.

Object Storage Use Cases

There are numerous use cases for object storage, thanks to its scalability, flexibility, and ease of use. Some of the most common use cases include:

Backup and archiving
Object storage is an excellent choice for storing backup and archive data, thanks to its durability, scalability, and cost-effectiveness. The ability to store custom metadata with each object allows organizations to easily manage retention policies and ensure compliance with relevant regulations.

Big data analytics
The horizontal scalability and programmability of object storage make it a natural choice for storing and processing large volumes of unstructured data in big data analytics platforms. Custom metadata schemes can be used to enrich the data and enable more advanced analytics capabilities.

Media storage and delivery
Object storage is a popular choice for storing and delivering media files, such as images, video, and audio. Its scalability and performance make it well-suited to handling large volumes of media files, while its support for various data formats and access methods enables seamless integration with content delivery networks and other media delivery solutions.

Internet of Things (IoT)
As the number of connected IoT devices continues to grow, so too does the amount of data they generate. Object storage is well-suited to handle the storage and management of this data, thanks to its scalability, flexibility, and support for unstructured data formats.

How to Choose an Object-Based Storage Solution

When choosing an object storage solution, there are several factors to consider. Some of the most important factors include:

  • Scalability: One of the primary strengths of object storage is its ability to scale horizontally, so it’s essential to choose a platform that can grow with your organization’s data needs. Look for a solution that can easily accommodate massive amounts of data without sacrificing performance or manageability.
  • Data durability and protection: Ensuring the integrity and availability of your data is critical, so look for an object storage platform that offers robust data protection features, such as erasure coding, replication, or versioning. Additionally, consider the platform’s durability guarantees – how likely is it that your data will be lost or corrupted?
  • Cost: Cost is always a consideration when choosing a storage solution, and object storage is no exception. Be sure to evaluate the total cost of ownership (TCO) of the platform, including factors such as hardware, software, maintenance, and support costs. Additionally, if you’re considering a cloud-based solution, be sure to factor in the costs of data transfer and storage.
  • Performance: While object storage is not typically designed for high-performance, low-latency workloads, it’s still important to choose a platform that can deliver acceptable performance for your organization’s specific use cases. Consider factors such as throughput, latency, and data transfer speed when evaluating performance.
  • Integration and compatibility: The ability to integrate the object storage platform with your existing infrastructure and applications is essential. Look for a solution that supports industry-standard APIs and protocols, as well as compatibility with your organization’s preferred development languages and tools.

See Additional Guides on Key Data Security Topics

Together with our content partners, we have authored in-depth guides on several other topics that can also be useful as you explore the world of data security.

EDR

Authored by Cynet

Ransomware Protection

Authored by Cynet

AWS SQL

Authored by NetApp

Learn More About Object Storage

Object Storage vs. File Storage: What’s the Difference?

Object Storage vs. Block Storage: What’s the Difference?

6 Best Practices for Object Storage Deployment

How Object Storage Protects You From Ransomware

Enhancing Object Storage Analytics: Adding Metadata Labels to S3 Images with TensorFlow

S3 Compatible Storage Solutions Compared

Understanding Cloud Native Storage

Object Storage in the Cloud: 4 Providers Compared

Cloudian S3 Compatible Enterprise Object Storage – Watch 1 min Overview Video

S3 Compatible Storage Solutions Compared

S3 Compatible Storage, On-Prem

Today’s emerging on-prem enterprise storage medium is S3 compatible storage. Initially used only in the cloud, S3 storage is now being extended to on-prem and private cloud deployments.

The term “S3 compatible” means that the storage employs the S3 API as its “language.” Applications that speak the S3 API should be able to plug and play with S3 compatible storage.

A growing number of applications now support this storage type, thus benefitting from its unique attributes:

  • Scale: Designed to grow limitlessly within a single namespace
  • Geo-distribution: A single storage system can span multiple sites
  • Cost: Purpose-built to run on industry-standard servers, thus benefitting from the volume and efficiencies of that industry
  • Reliable data transport: The only storage type invented in the age of the Internet, S3-compatible storage is built to manage and move massive data volumes over WANs

Cloudian specializes in S3-compatible storage, but other examples of applications and devices the now employ S3 are Rubrik, Veeam, Commvault, Splunk, Pure Storage, Adobe, VERITAS, Hadoop, NetApp, EMC, Komprise, and more.

This is part of an extensive series of articles about S3 Storage.

Clarifying the Terms

But what is S3-compatible storage? This storage type goes by multiple names and can also be called:

Object storage: The underlying technology for S3 compatible storage is object storage. Over the years, multiple APIs have been used to access object storage, but the S3 API is now the most common.

Cloud storage: Most large-scale cloud storage today is object storage, and most of it employs the S3 API. There are multiple ways of referring to essentially the same thing: S3-compatible storage.

Benefits of S3 Compatible Storage On-Prem

There are 5 key reasons to deploy S3 compatible storage in your data center:

  1. Scale: S3-compatible solutions are designed to scale in a single namespace, and without disruption, to an exabyte. Grow your storage without adding workload.
  2. 70% less cost than public cloud: With industry-standard hardware, these solutions deliver the greatest value: less cost per GB and higher density. Also, no ingress/egress fees.
  3. Performance: Hardware is in your data center for low latency and high bandwidth.
  4. Control: Data is behind your firewall, so you consistently apply security and control access.
  5. Cloud compatibility: S3 is compatible with cloud storage, so you can employ cloud when you need it, without disruption. Capitalize on the growing ecosystem of S3 compatible applications. Seamlessly move data and applications from on-prem to cloud.

The S3 API

S3 compatible storage is built on the Amazon S3 Application Programming Interface, better known as the S3 API, the most common way in which data is stored, managed, and retrieved by object stores. Originally created for the Amazon S3 Simple Storage Service (read about the API here), the widely adopted S3 API is now the de facto standard for object storage, employed by vendors and cloud providers industry-wide.

Not All S3 Compatible Storage APIs Are Equal

Compared with established file protocols such as NFS, the S3 API is relatively new and rapidly evolving. Among object storage vendors, S3 API compliance varies from below 50% to over 90%. This difference becomes material when an application — or an updated version of that app— fails due to S3 API incompatibility.

Cloudian is the only object storage solution to exclusively support the S3 API. Launched in 2011, Cloudian’s many years of S3 API development translate to the industry’s highest level of compliance.

Employing the S3 API makes an object storage solution flexible and powerful for three reasons:

1) Standardization in S3 Compatible Storage

With Cloudian, any object written using the S3 API can be used by other S3-enabled applications and object storage solutions; the existing code works out of the box.

S3 compatible storage software

2) Maturity 

The S3 API provides a wide variety of features that meet virtually every need for an object store. End users planning to deploy object stores can access the plentiful resources of the S3 community — both individuals and companies.

3) Rich Feature Set

The S3 API is the only storage “language” created in the era of the internet. The other common storage protocols (SMB and NFS) were created prior to the internet’s meteoric growth, and therefore did not factor in the needs of this infrastructure. As a result, only the S3 API includes features such as multi-part upload that make it easy to reliably transfer large files over dodgy WAN links.

 

The Cloudian Difference

Among the S3 compatible storage vendors, only Cloudian HyperStore was built from the start on the S3 API.

Cloudian S3 compatible storage API is designed into the Cloudian storage layer

 

Translation Layers Introduce Potential Compatibility Challenges

Competitive solutions employ a translation layer (or some sort of “access layer” or software gateway), which introduces the risk of compatibility challenges. Cloudian has no translation layer, hence we refer to it as “S3 Native.”

Translation layer leads to incompatibility

Cloud Storage in the Data Center

The combination of object storage and the de facto language standard now creates the option for cloud-connected storage in the data center. For the cloud, AWS has set the standard with the S3 Storage Service. Now data center managers can capitalize on that identical set of capabilities in their own data center with Cloudian S3 compatible storage.

See the S3 API at Work

The City of Montebello uses the S3 API as a mechanism for streaming live video from busses to a central monitoring facility where it is recorded and stored with metadata to assist with search.

MSPs Look to Cloudian to Help Grow S3-Compatible Object Storage Business

In cloud storage, there are hundreds of managed service providers (MSPs) who offer enterprise-grade, S3-API storage services. Increasingly, these MSPs look to Cloudian as the supplier for their underlying object storage infrastructure.

Local and regional MSPs offer their customers great value in several ways. They act as a trusted guide for large companies, and they provide rich, turnkey service offerings for smaller ones. Furthermore, their regional focus builds deep connections with local businesses as well as expertise in compliance and regulatory issues, benefits that remove technical barriers, reduce risk, and accelerate cloud adoption.

A wide range of high-value services fall within their offerings:  storage, archive, collaboration, and data protection, among others. These services require cost competitive storage, with scalability that is both limitless and painless. They also require S3 interoperability that ensures trouble-free operation and high customer satisfaction. That’s where Cloudian HyperStore comes in.

Over the past year, we’ve been pleased to help dozens of global MSPs deploy and manage successful cloud storage offerings, building profitable businesses on Cloudian’s infinitely scalable platform.

Why MSPs Choose Cloudian

Cloudian object storage uniquely addresses this market with MSP-oriented features, including:

  • 100% native S3 API: Ensures trouble-free operation with best-in-class interoperability
  • Quality of Service (QoS): Eliminates the “noisy neighbor” problem with bandwidth controls
  • Multi-tenancy: Unique namespace for each client
  • Branded UI: Provides customers with a management portal that highlights your brand
  • Configurable data protection: Lets customers configure data protection settings to meet their specific needs, rather than one-size-fits all

Additionally, our object storage platform contains a full suite of other features that bring key advantages to MSPs:

  • Start small and grow: HyperStore allows MSPs to scale with demand. Start with a few dozen terabytes and scale up to hundreds of petabytes and beyond — with zero disruption
  • Appliance or Software: Deploy Cloudian HyperStore as a pre-configured appliance — with full support — or as software on your server.
  • Geo-distribution: You can deploy Cloudian HyperStore in a single data center, or distribute across multiple data centers with data replication to ensure uninterrupted service, even in the event of a data center outage
  • File services: Deploy NFS/SMB file services for additional value-add

 

A Growing List of MSP Customers

A sample of the MSPs now offering Cloudian-based includes:

Learn more about the value these MSPs saw by checking out our solutions.

 

Object Storage Bucket-Level Auto-Tiering with Cloudian

As discussed in my previous blog post, ‘An Introduction to Data Tiering’, there is huge value in using different storage tiers within a data storage architecture to ensure that your different data sets are stored on the appropriate technology. Now I’d like to explain how the Cloudian HyperStore system supports object storage ‘auto-tiering’, whereby objects can be automatically moved from local HyperStore storage to a destination storage system on a predefined schedule based upon data lifecycle policies.

As discussed in my previous blog post, ‘An Introduction to Data Tiering’, there is huge value in using different storage tiers within a data storage architecture to ensure that your different data sets are stored on the appropriate technology. Now I’d like to explain how the Cloudian HyperStore system supports object storage ‘auto-tiering’, whereby objects can be automatically moved from local HyperStore storage to a destination storage system on a predefined schedule based upon data lifecycle policies.

Cloudian HyperStore can be integrated with any of the following destination cloud storage platforms as a target for tiered data:

  • Amazon S3
  • Amazon Glacier
  • Google Cloud Platform
  • Any Cloud service offering S3 API connectivity
  • A remotely located Cloudian HyperStore cluster

Granular Control with Cloudian HyperStore

For any data storage system, granularity of control and management is extremely important –  data sets often have varying management requirements with the need to apply different Service Level Agreements (SLAs) as appropriate to the value of the data to an organisation.

Cloudian HyperStore provides the ability to manage data at the bucket level, providing flexibility at a granular level to allow SLA and management control (note: a “bucket” is an S3 data container, similar to a LUN in block storage or a file system in NAS systems). HyperStore provides the following as control parameters at the bucket level:

  • Data protection – Select from replication or erasure coding of data, plus single or multi-site data distribution
  • Consistency level – Control of replication techniques (synchronous vs asynchronous)
  • Access permissions – User and group control access to data
  • Disaster recovery – Data replication to public cloud
  • Encryption – Data at rest protection for security compliance
  • Compression – Reduction of the effective raw storage used to store data objects
  • Data size threshold – Variable storage location of data based upon the data object size
  • Lifecycle policies – Data management rules for tiering and data expiration

Cloudian HyperStore manages data tiering via lifecycle policies as can be seen in the image below:

Auto-tiering is configurable on a per-bucket basis, with each bucket allowed different lifecycle policies based upon rules. Examples of these include:

  1.      Which data objects to apply the lifecycle rule to. This can include:
  • All objects in the bucket
  • Objects for which the name starts with a specific prefix (such as prefix “Meetings/2015/”)
  1.      The tiering schedule, which can be specified using one of three methods:
  • Move objects X number of days after they’re created
  • Move objects if they go X number of days without being accessed
  • Move objects on a fixed date — such as December 31, 2016

When a data object becomes a candidate for tiering, a small stub object is retained on the HyperStore cluster. The stub acts as a pointer to the actual data object, so the data object still appears as if it’s stored in the local cluster. To the end user, there is no change to the action of accessing data, but the object does display a special icon denoting the fact that the data object has been moved.

For auto-tiering to a Cloud provider such as Amazon or Google, an account is required along with associated account access credentials.

Accessing Data After Auto-Tiering

To access objects after they’ve been auto-tiered to public cloud services, the objects can be accessed either directly through a public cloud platform (using the applicable account and credentials) or via the local HyperStore system. There are three options for retrieving tiered data:

  1.      Restoring objects –   When a user accesses a data file, they are directed to the local stub file held on HyperStore which then redirects the user request to the actual location of the data object (tiered target platform).

A copy of the data object is restored back to a local HyperStore bucket from the tiered storage and the user request will be performed on the data object once copied back. A time limit can be set for how long to retain the retrieved object locally, before returning to the secondary tier.

This is considered the best option to use when accessing data relatively frequently and you want to avoid any performance impact incurred by traversing the internet and any access costs applied by service providers for data access/retrieval. Storage capacity must be managed on the local HyperStore cluster to ensure that there is sufficient “cache” for object retrievals.

  1.      Streaming objects – Streams data directly to the client without restoring the data to the local HyperStore cluster first. When the file is closed, any modifications are made to the object in situ on the tiered location. Any metadata modifications will be updated in both local HyperStore database and on the tiered platform.

This is considered the best option to use when accessing data relatively infrequently and concern about the storage capacity of the local HyperStore cluster is an issue, but performance will be lower as the data requests are traversing the internet and access costs may be applied by the service provider every time this file is read.

  1.      Direct access – Objects auto-tiered to public cloud services can be accessed directly by another application or via your standard public cloud interface, such as the AWS Management Console. This method fully bypasses the HyperStore cluster. Because objects are written to the cloud using the standard S3 API, and include a copy of the object’s metadata, they can be referenced directly.

Storing objects in this openly accessible manner — with co-located rich metadata — is useful in several instances:

  1. A disaster recovery scenario where the HyperStore cluster is not available
  2. Facilitating data migration to another platform
  3. Enabling access from a separate cloud-based application, such as content distribution
  4. Providing open access to data, without reliance on a separate database to provide indexing

HyperStore provides great flexibility for leveraging hybrid cloud deployments where you get to set the policy on which data is stored in a public or private cloud. Learn more about HyperStore here.

 

YOU MAY ALSO BE INTERESTED IN

Object Storage vs. Block Storage: What’s the Difference?

An Introduction to Data Tiering

All data is not equal due to factors such as frequency of access, security needs, and cost considerations, therefore data storage architectures need to provide different storage tiers to address these varying requirements. Storage tiers differ depending on disk drive types, RAID configurations or even completely different storage sub-systems, which offer different IP profiles and cost impact.

Data tiering allows the movement of data between different storage tiers, which allows an organization to ensure that the appropriate data resides on the appropriate storage technology. In modern storage architectures, this data movement is invisible to the end-user application and is typically controlled and automated by storage policies. Typical data tiers may include:

  1. Flash storage – High value, high-performance requirements, usually smaller data sets and cost is less important compare to the performance Service Level Agreement (SLA) required
  2. Traditional SAN/NAS Storage arrays – Medium value, medium performance, medium cost sensitivity
  3. Object Storage – Less frequently accessed data with larger data sets. Cost is an important consideration
  4. Public Cloud –  Long-term archival for data that is never accessed

Typically, structured data sets belonging to applications/data sources such as OLTP databases, CRM, email systems and virtual machines will be stored on data tiers 1 and 2 as above. Unstructured data is more commonly moving to tiers 3 and 4 as these are typically much larger data sets where performance is not as critical and cost becomes a more significant factor in management and purchasing decisions.

Some Shortcomings of Data Tiering to Public Cloud

Public cloud services have become an attractive data tiering solution, especially for unstructured data, but there are considerations around public cloud use:

  1. Performance – Public network access will typically be a bottleneck when reading and writing data to public cloud platforms, along with data retrieval times (based on the SLA provided by the cloud service). Especially for backup data, backup and recovery windows are still incredibly important, so for the most relevant backup sets it is worth considering to hold onsite and only archive older backup data to the cloud.
  2. Security – Certain data sets/industries have regulations stipulating that data must not be stored in the cloud. Being able to control what data is sent to the cloud is of major importance.
  3. Access patterns – Data that is re-read frequently may incur additional network bandwidth costs imposed by the public cloud service provider. Understanding your use of data is vital to control the costs associated with data downloads.
  4. Cost – As well as bandwidth costs associated with reading data, storing large quantities of data in the cloud may not make the most economical sense, especially when compared to the economics of on-premise cloud storage. Evaluations should be made.

Using Hybrid Cloud for a Balanced Data Tier Strategy

For unstructured data, a hybrid approach to data management is key with an automation engine, data classification and granular control of data necessary requirements to really deliver on this premise.

With a hybrid cloud approach, you can push any data to the public cloud while also affording you the control that comes with on-premises storage. For any data storage system, granularity of control and management is extremely important as different data sets have different management requirements with the need to apply different SLAs as appropriate to the value of the data to an organization.

Cloudian HyperStore is a solution that gives you that flexibility for easily moving between data tiers 3 and 4 listed earlier in this post. Not only do you get the control and security from your data center, you can integrate HyperStore with many different destination cloud storage platforms, including Amazon S3/Glacier, Google Cloud Platform, and any other cloud service offering S3 API connectivity.

Learn more about our solutions today.

Learn more about NAS backup here.

 

New HyperStore 4000: Highest density storage

Rack space and budget. Most data centers are short on both. Yet somehow, you’re expected to accommodate a 50% increase in unstructured data volume annually. That’s a problem.

The new solution is the HyperStore 4000. With 700TB in just 4U of rack height, it’s nearly 2X the density of our earlier models. And it delivers storage in your data center at prices on par with the public cloud: about ½ cent per GB per month. Space savings and cost savings in one.

The HyperStore 4000, by the numbers

The HyperStore 4000 appliance was built to handle massive amounts of data. It’s housed in a 7” high 4U enclosure, with 2 nodes and a max capacity of 700TB. The drive sizes range from 4TB to 10TB and has 256GB of memory (128GB per node).

Better yet, the appliance reduces storage costs by 40% (versus other Cloudian solutions) – data management now costs half a cent per GB per month. Even with the reduced cost, there is no drop in data availability – with a three-appliance cluster, you’ll still see 99.999999% data durability.

For the full list of specs, check out our datasheet.

Save time, your scarcest commodity

With most storage systems, as your needs grow, your management headaches grow with them. But the HyperStore 4000 grows painlessly. You just add nodes. No disruption and no data migration.

We specifically designed the HyperStore 4000 appliance to help customers in industries with huge data needs such as life sciences, healthcare, and entertainment. These are the industries where data growth is exploding every year and where modern data centers feel the most burden, as they need high-density storage with peak performance for data protection, video surveillance, research, archival, and more. Now you can meet these growing needs without growing pains.

Finally, the HyperStore 4000 has a 100% native S3 API, and has the industry’s highest level of S3 API compatibility. In fact, we guarantee it to work with your S3-enabled applications.

Be sure to also take a look at Cloudian’s other solutions to see which one is right for you.

Cloudian Customer Receives Commvault Innovation Award

Cloudian Customer Receives Commvault Innovation Award for Data Protection with Object Storage

A Cloudian customer, Schuberg Philis, has been recognized by Commvault for their innovation in a data protection deployment with Cloudian object storage. Several aspects of this award-winning solution illustrate advancements that make backup a very exciting topic right now:

  • Object storage as a target: On-premises S3-compatible storage is the backup target in this solution
  • Backup as a service: 3800 clients employ this environment
  • Local and remote backup: Clients being protected are both local (within the Schuberg facility) and remote

Data protection is alive with innovation, and this illustrates why. Data center managers now have more options than ever to reduce headaches, cut costs, and increase service levels.

Object storage helps by providing a seamlessly scalable backup target that a) works with most backup solutions, including Commvault, b) delivers disk performance at costs approaching tape, and c) includes a broad range of capabilities including compression, encryption, and deduplication.

Backup as a service is now more practical than ever, thanks to the S3 protocol that enhances data delivery over network connections.

Schuberg Philis brought these innovations together to offer Data Management as a Service (DMS). This is a multi-tenant data protection solution that’s based on Commvault software and Cloudian storage. It runs within Schuberg Philis’ Mission Critical Cloud Infrastructure.

As a centralized backup and restore platform, DMS includes a wide swath of features such as object storage, SQL AlwaysOn, clustering, and encryption. These features make it easier for customers to manage data protection options without sacrificing data integrity. Commvault took notice and awarded Schuberg Philis a global Service Provider Innovation Award.

We’re very proud that we could be a part of this great solution!

To learn more about the economics of object storage, read this Object Storage Buyer’s Guide. Learn how you too can save a bundle, and beat your SLAs, all with the backup software you already have.

Better Backup With the Software You Already Have

You know the challenges of the backup process. Veritas and Commvault are good products, but backup is still a chore. Your three choices for a backup target all have challenges: Tape is troublesome, disk is expensive, and backup to the cloud is slow.

How to save cost, reduce stress, and keep using the software you already know

The New Backup Target: Hybrid Cloud

You know the challenges of the backup process. Veritas and Commvault are good products, but backup is still a chore. Your three choices for a backup target all have challenges: Tape is troublesome, disk is expensive, and backup to the cloud is slow.

As an IT manager, you pick the best solution you can afford, but you’re often forced to make compromises along the way. Too often, the result is busted backup windows and unmet RTO and RPO SLAs, not to mention hours of wasted time and accumulated stress.

Now there’s a fourth backup target option:  Hybrid Cloud. (see Backup Solutions Note)

Hybrid cloud as a target gives you a faster, more reliable, lower cost process — free of capacity constraints. It works right now with the software you already know. And you can get started at zero upfront cost.

How the Hybrid Cloud Helps

Hybrid cloud integrates an on-premises disk-based target with a cloud-based target. Both the on-prem storage and cloud storage use the same interface and are managed as a single storage pool.

Their respective functions are:

  • On-prem target: Fast disk-backup. Provides predictable backup time; ensures immediate access for RTO/RPO SLAs
  • Public cloud target: DR repository; low-cost and offsite, it provides the ideal long-term archive, plus overflow capacity for limitless scalability

Works with Existing Backup Software

Backup procedures are proven through years of development. And you know well the software you have. The hybrid cloud approach leverages all of that investment and learning by preserving your existing processes.

To the backup software, the hybrid cloud appears exactly as cloud storage. (Connectors to Amazon S3 and other services are now available with most popular backup software.)

With hybrid cloud, that connector is simply directed at the on-prem storage. The on-prem storage then connects to the cloud. The two are managed as a single, limitlessly scalable storage pool.

The on prem S3-compatible storage is then directed at the S3 public cloud for data tiering purposes. The most recent backups — ie, the ones you’re most likely to use — are kept on prem. The older copies are migrated to the cloud.

The combined solution becomes a simple, drop-in replacement for existing backup target technologies. The result: on-site storage for fast access, and cloud storage for low-cost archive and DR.

In summary, hybrid combines a petabyte-scalable, high-performance on-premises backup target with seamless cloud storage integration. Together they let you retain a familiar workflow while ensuring success on the objectives that matter to you: backup window predictability, and repeatable RTO / RPO.

Start Small and Grow

Best of all, you can start with a small deployment, prove it out, and grow. On-prem S3 storage can be deployed on servers you already have, or deployed as preconfigured appliances.

There are even zero-upfront-cost options using Amazon metered-by-use software from the Amazon Marketplace.

Eight Ways Hybrid Cloud from Cloudian Makes Backup Better

Cloudian is the on-prem storage node in a hybrid storage configuration. It features the industry’s highest level of S3 compatibility, ensuring full interoperability with Veritas, Commvault, and Rubrik.

The Cloudian architecture is a scale-out storage cluster comprised of shared-nothing storage nodes. Your media servers connect to the on-prem Cloudian cluster via Ethernet and communicates via an S3-compatible API. Your backup software views the cluster exactly as it views cloud storage. It stores data to Cloudian exactly as it would to cloud storage.

The difference with Cloudian vs cloud alone is that all recent backups are stored locally for quick recovery when needed. Policy-based migration then allows older snapshots to be migrated to the public cloud. This frees up local capacity, and also provides an offsite copy for DR use.

Here are eight ways this helps:

1) Performance to handle the largest environments

Cloudian scales to petabytes with a scaling model that grows in both capacity and bandwidth. Predictable backup windows result from Cloudian’s high streaming bandwidth: Writes in excess of 5000 MB/s can be achieved, or 18TB per hour.

2) Petabyte-scalable

You can start small with just three nodes, and scale to petabytes simply by adding nodes. Scaling is seamless and does not require downtime.

3) 70% less cost than conventional disk

Built on industry-standard hardware, Cloudian drives down the cost of on-prem, disk-based storage to 1¢/GB/month or less, depending on capacity.

4) Manage one data pool

Cloudian maintains data in a single pool across all nodes. You get one-to-many auto-replication, enhancing data durability. No need to juggle what’s “active” or “passive,” create complex policies and snapshot management techniques, or track which sites are replicating to where.

5) Distributed architecture for global data protection

Enterprises struggle to manage backup at remote offices. With Cloudian, clustered nodes can be deployed globally and interconnected, thus allowing data to be automatically replicated across sites.  Because the nodes form a single namespace, you can implement policy-based data migration to the cloud for DR purposes. You get global data protection with fast local recovery, all managed from a single location.

6) Deploy as appliances, or on your own servers

Cloudian is built on industry-standard hardware. You have the flexibility to buy either pre-configured, fully supported appliances, or software for installation on the servers you choose. Either way, you benefit from the value of commodity hardware.

7) Drop-in integration

Cloudian can be immediately integrated with backup software packages that support cloud storage, including Veritas NetBackup, Veritas Backup Exec, Commvault Simpana and Rubrik. Cloudian is viewed exactly as cloud storage for both backup and recovery. For information that has been migrated to the cloud, Cloudian transparently retrieves that data and presents it to the media server.

8) Start small, even at zero upfront cost

Contact Cloudian to get started. We can even show you options that get you started at zero upfront cost, with Cloudian from the Amazon Marketplace.

For more information, read the Backup Solutions Note or specific data protection solutions with

Configuration Guides for Veritas and Commvault are also available.

  • Veritas
  • Commvault

Why Internet Unie Chose Cloudian for Hybrid Cloud Storage

Internet Unie, a service provider in the Netherlands, has recently deployed an innovative hybrid cloud service, combining Cloudian object storage in their data center together with Amazon S3 storage.

The new service allows their colocation customers to employ local S3 storage in their data center, with additional capacity available in the AWS public cloud.

Why would a service provider launch a service that employs another service provider (in this case Amazon)?

The answer is simple: it fills a real business need and gives Internet Unie a competitive advantage.

By offering their customers this hybrid service, Internet Unie meets multiple possible requirements:

    • Performance: Local storage provides cloud-compatible capacity without the latency of a long network hop
    • Data governance: Locally stored data does not leave the data center
    • Capacity flexibility: Data can be tiered off to the cloud when desired, meaning capacity is always there
    • Disaster recovery: Backup information can be moved off site at any time
    • Cost: Locally stored information costs nothing to access, meaning that cloud storage invoices become far more predictable
    • Archival storage: Cloud archival services are very cost effective for rarely accessed information
    • Business simplicity: One invoice for both on prem and cloud storage, thanks to the Amazon Marketplace metered-by-use program

Internet Unie summed it up this way:

“This hybrid service opens up enormous possibilities for those using the AWS service cloud offerings and need to store certain data types in a private cloud, for reasons such as data governance policies. With Cloudian’s new offering on AWS, our customers can point their applications to either cloud storage or on-premises storage, and it’s completely transparent,” said Arvid Cauwels, Sales Director at Internet Unie. “With AWS metering now available for Cloudian storage, customers get one AWS invoice for both their public and private cloud storage usage.”

Cloudian is a natural fit due to our native support for the Amazon S3 API, which makes it easy to tier between a Cloudian storage system in the Internet Unie data center and AWS cloud storage. Additionally, Cloudian supports AWS metering, which pulls all usage and billing (for both public and private cloud) into a single monthly AWS invoice.

Hybrid cloud represents a ‘best of both worlds’ solution, giving customers extra flexibility and control while providing limitless scalability. Read our blog post to learn more about why you should consider a hybrid cloud solution.

 

AWS re:Invent Attendees See Benefits of Hybrid Storage

AWS re:Invent was a fantastic show this year. The show has seen phenomenal growth, with over 32,000 attendees, up from 18,000 attendees last year.

AWS re:Invent

Many visitors were looking for solutions to let them integrate their on-premises operations with the cloud. By adopting a hybrid cloud storage approach, they would be able to capitalize on the scalability and cost of cloud storage when appropriate, while also maintaining the cost predictability and control of on-prem storage.

For these visitors, Cloudian proved to be the perfect fit. We provide 100% native Amazon S3 object storage, with automated tiering between the data center and the cloud. Our HyperStore solution is also available directly from AWS Marketplace, which means users can get all their usage and billing data within a single monthly invoice from AWS.

Steve Varner, Principal Data Engineer at Motorola Solutions, visited our booth and had this to say afterwards:

Steve Varner

Interested in learning more about Cloudian? Contact us or try it out for yourself.

 

Categories