SDK7 - Under the hood: CRDT & Protobuf
Let’s talk a little about two of the main axes of SDK7 that will significantly improve the quality and the performance of the platform for content creators and final users.
With SDK7 adopting ‘protobuf’ instead of JSON -mainly due to scalability issues-, messages won’t be -strings- anymore and will turn to be binary encoded, meaning they will become a more compact format and easier to serialize/deserialize for the different terminals. On the other hand, the implementation of ‘CRDT’ protocol technology will allow processing simultaneous messages -no matter the order of them- with a better preservation of the “user’s intention”, ensuring more logical and consistent final outputs, and their propagation.
CRDT
A catching phrase that I love to use to describe what happens with the implementation of a CRDT protocol is that “you can’t have a conflict because there’s no such thing as a conflict”.
So, a Conflict-free Replicated Data Type (CRDT) resolves conflicts without any special transformation and, as we said last time, this protocol is already developed for Decentraland SDK7. The main issue that CRDT tackles is the synchronization of state.
The problem with past messaging algorithms is that, to order and reproduce multiple individual instructions coming from different terminals and to show the same results as fast as possible in all of them, we’ll find some limitations that directly affect the UX. Also, messages from the scene to the rendering engine required to be ordered and complete, that generates a high level of coupling between the scene and the renderer, because both the scene and the system require synchronous messaging across all the lifecycle of the scene.
To implement the CRDT protocol, SDK7 has to be launched as a whole, but we can outline the lower layers on which CRDT will be deployed, and the upper layers that will be assembled based on this technology: the algorithm will be used for the network between peers, but also in the communication between the scene and the renderer, so the CRDT will live at the ECS level but also at the renderer.
The best illustration that we can use to demonstrate how CRDT operates and how it directly impacts UX, is a simple interactive scene with a door that can be opened and closed, and how this is reproduced equally in each player’s instance thanks to the enhancement of state synchronization.
Now that the basics from CRDT are covered, here is how it is used:
The entities and components of the scenes were serialized and sent to the renderer requiring all messages to arrive ordered and also depending on having all the messages being delivered. That made things easy to the renderer, assuming no message will ever be lost.
With CRDTs, we can ensure that a consistent state will be reached, no matter the order of the messages, as long as they arrive. That enables one of the big new features of the SDK7: decoupling the scene runtime from the renderer. The chosen CRDT is a ‘keyed-LWW-Element-Set’. In plain english, it is a map of key-values, and what travels as a package is the combination of a key and a value followed by a timestamp. The conflict-free resolution algorithm ensures that all actors will discard and accept the same packages based on their timestamp and content, to reach a consistent and conflict-free final state.
Now, what are those key-value combinations?
The key is composed of the entityId + componentId.
And the value is the information of the component e.g. Position of an entity, or its material. The CRDT protocol doesn’t know about what is in those values, treats them like a blob of bytes. But the new ECS and the Renderer do know how to interpret that blob of bytes. In most cases, that blob of bytes is serialized with protobuf.
Protobuf
Protocol buffer is actually not a protocol definition: we could say it is a schema definition language that allows us to establish our own protocols and enables us to serialize/deserialize in a more compact format -binary protocol-, and it’s meant for networking and storage. And more importantly, protobuf enables the evolution of the schemas (and their binary representations) while keeping backwards compatibility.
Why protobuf? JSON is good, but just not good enough for a platform like Decentraland due to performance issues. If your data is mainly string, then JSON format might be a good choice since it’s more human-readable, but with the need to exchange huge amounts of data of different types, protobuf brings a more compact, lightweight and faster solution.
A schema is the definition of the serialized messages ‘./person.proto’ and it associates data types with field names, using integers to identify each field. By encoding the semantics through schemas we can define and enforce a set of rules. As protobuf is language-independent and platform-neutral, it’s designed to evolve and adapt because schemas can change, while maintaining backward compatibility, and also provides a set of tools that allows us to write code out of schemas.
The key takeaway of this schema is that the fields are defined by a Field number and a Field type (fig 1). Based on that definition, the protobuf tooling will generate optimized code for the specific readers and writers.
Another aspect to highlight from the binary format, is that there are no headers/trailers in the serialization (fig 2), which may lead to complicated and stateful parsers (like JSON). Instead of that, protobuf uses prefixed sizes to read and write the next values, making it easier to read and deserialize.
Also, as we pointed out, protobuf is not just a message format, it also incorporates a set of rules to define and exchange messages thanks to their binary format, and it supports a wider range of data/wire types than JSON. Thanks to the way those types are encoded in the messages, there is no need for readers or writers to know the full schema of the messages, enabling easier upgrades to the protocol and backwards compatibility.
So, how does protobuf work?
To conclude…
These improvements in the messaging system will inevitably lead to an overall performance enhancement, mainly around the synchronization issue, and to bring more and better experiences in multiplayer content. CRDT and protobuf are solid fundamentals for the SDK, the platform, and a great tool to overcome possible and existing scalability issues.
If you want to explore or go further on this topics, here you can find some useful links:
- keyed-LWW-Element-Set
- gRPC
- RFC - Comms protocol for explorers
- Protocol buffers
- Protocol definitions for the new SDK
Again, you’re invited and welcome to share your thoughts. Next time we’ll try to take an approach on data oriented programming and entities!