The growth of big data and IoT has exposed the bottlenecks and failures of traditional storage methods, especially with regards to unstructured data. Object storage addresses these issues, while also providing flexibility, cost-effectiveness, security, and availability—making it ideally suited for hosting unstructured data.
With object storage, data is managed by creating a unique identifier for each item stored. Unlike hierarchically stored data, object storage enables the number of stored items to grow beyond the limits of traditional storage systems, while still maintaining the integrity and consistency of the data.
Object stores also protect every piece of data placed on them by design. No other backup or copy of the data needs to occur. Intrinsically, objects are fully protected, and every object storage system provides protection using two of three possible methods:
- Redundant Nodes
All object storage systems use a multi-node system. Each “node” is a rack-mounted server filled with internal disks. None of the disks use RAID protection. Each disk is formatted and controlled by the object store as an individual place to store objects. By design, nodes are spread out across multiple data centers to provide protection in case of complete site failure.
- Full Object Copy
Some object stores take an object and protect it by making three copies of the object and placing the copies on different nodes. Any individual disk failure or complete node failure is a non-event as there are always two other copies of the object available. A background scan is always running across the object store system to verify the integrity of each object, and if it identifies any damage or missing data, it’ll build a new copy of the object.
With some object stores, you can change the protection level to a number of copies other than three.
Other object stores use an erasure-code, an advance mathematical process, to protect data. Here’s how it works:
- An object is run through a math process to increase its size with redundant data from the object itself.
- The inflated object is sliced into multiple pieces.
- The different pieces are scattered across all the nodes—and individual disks within each node—in the system.
The goal of these steps is to provide a protection scheme that can be described as #/#/#. A very small protection scheme might be 5/7/13. There are different protection schemes available that use larger numbers. However, in this example:
- The object was expanded and sliced into thirteen pieces.
- Seven of the thirteen pieces are needed to have a read-write version of all the data in the object.
- Only five of the thirteen pieces are needed to have a read-only copy of the data.
By copying the “slices” to as many geographically dispersed nodes as possible, erasure-codes offer full protection against any disk, node, or site failure—while using storage that’s 160% to 180% of the original size of the files. By comparison, full object copy protection, in the normal three-copy mode, takes 300% of the original size of the files.
But like full object copy, erasure-code protection includes a scan process running in the background to ensure the integrity of each slice, to verify that all of the slices are present, and to rebuild missing slices as necessary.
We’ll continue to explore the inherent benefits of object storage, so come back soon.