Welcome back for the second part of this introductory series on ThreeFold Grid technology. Last time we took a high level view of all that the Grid offers. This time, weâll be zooming in on storage, covering the low level components and how they can provide a familiar experience like the cloud âdrivesâ and âboxesâ we use today.
The ThreeFold Grid offers several different storage primitives that can be combined in various ways to meet the needs of developers and end users of Grid capacity. Ultimately, all storage services are a way of presenting the underlying hardware to software workloads in various ways, according to the desired features. ThreeFold has created a few unique storage offerings, providing distinct benefits over legacy cloud infrastructure. From the start, every component has been designed for optimal efficiency and they work together to offer an unprecedented level of security.
Our most basic storage primitive is a âvolumeâ, which simply offers some disk space to an application in a generic way. Weâll focus more on the innovative solutions that are built on the Zero Database, or Zdb. These are compatible with existing use cases but also represent a quantum leap forward for privacy and security. They are called Zdbfs, Zstor, and QSFS. Letâs take a look at each component and how they work together to provide a complete set of tools for developers building on the Grid. Weâll also see why this matters to end users of services that are hosted on the Grid.
đZdb
Zdb is a low level offering that implements with a subset of features from the popular Redis protocol. It is a key-value store, which means that information is indexed like a dictionary. Each âwordâ in the database is associated with a âdefinitionâ or piece of data. Zdb is super fast and efficient, along with operating in an âappend onlyâ manner. This means that all new data is added to empty space following the end of existing data, never overwritten. Itâs like writing in pen on paper without leaving any white space.
Append only has many advantages, including extending the life of certain hardware and offering archiving features out of the box. The disadvantage, of course, is that old data is retained even when no longer needed. However, this can be addressed by periodically âcompactingâ the data, to remove whatâs no longer needed and restructure what remains with the same level of efficiency. We use Zdb as a base layer for more complex storage implementations that benefit from these features.
đZdbfs
While Zdb has all of those neat properties, it requires that applications utilize a specific database interface. To expand its capabilities, we have the Zdb Filesystem. With Zdbfs, the back end Zdb is exposed through the most common interface used for storing and retrieving data: a filesystem. This is the same format weâre all familiar with from the file explorers on our computers. When running Zdbfs, a new âfolderâ appears on the system, allowing reading and writing to the connected Zdb without any concern for the underlying database. With Zdbfs, nearly all existing tools for working with data can be plugged in to and benefit from the advantages of Zdb.
đZstor
So far, weâve been discussing solutions that utilize storage space on a single node only. While this offers flexibility for developers with a variety of needs, it is also limited in the sense that the failure of a single node would result in the loss of all data. With that in mind, we offer the Zstor solution for spreading data across multiple nodes in a way that provides both resilience against failures and unprecedented security.
Zstor takes a single file as input, to be stored among a number of nodes as specified in its configuration. The system is flexible and able to cater to different needs for performance, redundancy, and geographical distribution. Understanding what happens behind the scenes requires a little math, but Iâll provide as simple an explanation as I can. Zstor has a very unique property: it does not actually upload any of the userâs data to the back end nodes, but it is able to reconstruct the data later based on the descriptive information that is stored.
đThe magic of erasure encoding
Sounds like magic? Well, letâs explore briefly how this is possible. Zstor relies on a technology known as erasure encoding, which was originally designed to protect data against errors that sometimes occur in the normal operation of computer hardware. Rather than simply making copies of the data as backups, a clever scheme is used to efficiently offer the same benefit. With basic redundancy, four extra copies would be needed to accommodate four failures without a loss of data. Erasure encoding can accomplish the same failure tolerance using less than half the space needed for the original data.
While erasure encoding can be implemented by storing the data itself along with some extra data known as âparityâ, it can also be implemented without storing the original data at all. As a simplified example, letâs say that we want to store the number 13. First, we take each digit individually, 1 and 3. Next, we calculate 1 + 3 = 4 and 3 - 1 = 2. By storing 4 and 2, along with the instructions to reverse these calculations, we have everything needed to get back to 13. If we add one more equation, say 1 - 3 = -2, we can restore the original data using any two of the numbers weâve computed.
Each of our values would be stored on separate nodes, while the instructions to recombine them would be stored in yet another location. If an attacker were to compromise one of these nodes, they would only have a number thatâs meaningless without the other elements. If a single node fails, the original data can be restored using the remaining nodes, and we can add a new node to bring us back to our desired state. By tuning these values, additional security or redundancy can be achieved. We might decide that we want 9 of 10 values to be present for reconstruction, representing a high level of security. Alternatively, a 2 of 10 arrangement would provide high redundancy, tolerating the failure of 8 nodes.
đQuantum Safe Filesystem
With all of these components working together, we have ThreeFoldâs flagship storage solution, the Quantum Safe Filesystem or QSFS. It uses Zstor to backup data written to a Zdbfs. This provides the convenient and highly compatible filesystem interface along with the exceptional security and redundancy benefits of Zstor. We call it quantum safe, because even an attacker with a quantum computer would not be able to decode usersâ data, if they managed to hack into one of the back end nodesâalready an extraordinary feat given the exceptional security of Zero OS.
đBringing it home
So, weâve toured a good bit of technology which might be feeling a bit abstract at this point. Bringing this home, our front end experience with these technologies can feel no different than any of the âdrivesâ and âboxesâ we use to store our files in the cloud. In fact, ThreeFold has already developed a prototype file browser based on QSFS which has many cool features like editing documents and viewing media directly in the web interface. Itâs one piece of a full suite of solutions weâre excited to showcase and invite the community to test soon.
I hope you found this piece informative and approachable. Thanks for joining me to learn and explore the wonderful world of ThreeFold technology. Weâll cover more aspects of what makes it all tick in future parts of this series. Do you have questions or feel like chatting about whatâs possible with ThreeFoldâs technology? Weâd love to hear from you on our forum or in our Telegram group.