SDK7 - Under the hood: Data oriented programming (ECS)
Today I thought that it was a good idea to take an overview on another one of the biggest changeovers for content creators and its consequences. We’re talking about ‘data-oriented programming’ also known as ‘data-oriented design’, which brings a new and revolutionary approach in the way to treat entities.
Just to refresh, an Entity Component System (ECS) is a software architectural pattern mostly used in video game development for the representation of game world objects. An ECS comprises entities -basic unit for building everything in Decentraland scenes- composed from components of data, with systems which operate on the components of these entities. All visible and invisible 3D objects and audio players in your scene will each be an entity that holds components.
Now, what are we talking about when we say data-oriented programming?
We can say it’s an approach that focuses on treating data as the central element, with all else being organized around it, to either access or modify that data. When we consider the data from this point of view, data is mere facts that can be interpreted in whatever way necessary to get the output data in the format it needs to be.
Data-oriented design avoids wasting resources by never assuming the design needs to exist anywhere other than in paper. This approach gives up some of the human readability by leaving the problem domain in the design document, bringing elements of constraints and expectations into the act of transforming data, but stops the machine from having to handle human concepts at any data level by just that same action. It makes progress by providing a solution to the current problem through some high-level code controlling sequences of events and specifying schema in which to give temporary meaning to the data.
The main benefit of this approach is in optimizing the speed at which data can be read from memory, which is often the main bottleneck while running modern applications and games. Adopting this approach very often results in improvements that are very noticeable in terms of hardware resources and overall performance.
Where do we come from?
ECS6 has a distant approach from what we call a ‘pure-ECS’ (data oriented), and it offers a go-between the classic game-object/behavior pattern and the ECS pattern. In this, the developer can use game-objects and attach a special form of behavior that converts game-objects into Entities and certain behaviors into Components.
As a go-between, ECS6 is more centered around the problem definition and Objects are not real things but abstract representations of the context in which the problem will be solved. The objects manipulate the data needed to represent them without any consideration for the hardware or the real-world data patterns or quantities.
The current pipeline for ECS components is prepared for use cases that require a memory footprint and processing that may be too big for a component that’s supposed to hold a single field of data, giving as a result a need for high maintenance.
This is why object-oriented design allows you to quickly build up first versions of applications, allowing you to put the first version of the design document or problem definition directly into the code, and make a quick attempt at a solution. But this is also why an object oriented design could commit ‘crimes against the hardware’.
Other problems associated to this approach are:
- A poor way to handle rendering and simple element placement, leading to much higher complexity when interpreting locality of elements
- High maintenance and complex evolutionary development
- Imminent inertia in keeping the problem domain coupled with the implementation.
- Technical structure and the design for a project often changes so much that there is barely any section from the first draft remaining unchanged in the final implementation.
Where are we going?
The main difference between past approaches and the brand-new ECS7, is that now an entity itself has neither any data nor any code, it’s is an abstract concept not represented by any data structure being just an ID. Therefore there will no longer be anything like a “class Entity”, since all the data of the entities will be contained in the Components.
From now on, an Entity will become just an ID so, in other words, it will become a mere number that is used as a reference to group different components. It will no longer be necessary to separately create an entity and then add it to the engine, this will all be done in a single act.
A component will become a “dumb” data container without any executable functions, and each component can be assigned to an entity. So now, all your executable game logic code will be held in the Systems -pure and simple functions- and all your logic comes from here: a system might hold data which is relevant to the system itself, but no data about the entities it processes.
What does this mean? all the logic will be given through systems, components will be the data, and entities will be just an ID.
Impact on content creators and new syntax
Yes, content creators will need to learn and get used to this new approach. This won’t necessarily be harder, but it’s a different approach that needs to be learnt and adopted, and content creators might need some time to get familiar with it.
If you have been using entities without any extension, only the syntax will be different. The semantics will map 1-1 to the new ECS. On the other hand, if you used entities as game objects, then this new ECS will require new semantics, data oriented semantics.
What basically we need to remember, or one of the basic statements that we should always keep in mind from now on is that everything that has intrinsic logic will be a system or a part of it. As we mentioned, we moved from an object-centered approach, to now focus mainly on the data. One of the main consequences of this, is that syntax will change:
/**
* OLD SDK
*/
// Define a group of all entities with VelocityComponent
const myGroup = engine.getComponentGroup(VelocityComponent)
// Define the system
class MoveSystem implements ISystem {
// This function is executed on every frame
update() {
// Iterate over the entities in an component group
for (let entity of myGroup.entities) {
// get entity's Transform
let transform = entity.getComponent(Transform)
// get entity's VelocityComponent
let velocity = entity.getComponent(VelocityComponent)
// add the velocity to the transform
transform.position = transform.position.add(velocity.vel)
}
}
}
// Add system to engine
engine.addSystem(new MoveSystem())
/**
* SDK 7
*/
// Define the system
function MoveSystem() {
// Iterate over the entities with a VelocityComponent
for (const [entity] of engine.getEntitiesWith(VelocityComponent)) {
// get entity's Transform (mutable)
let transform = Transform.getMutable(entity)
// get entity's VelocityComponent (read only)
let velocity = VelocityComponent.get(entity)
// add the velocity to the transform
transform.position = Vector3.add(transform.position, velocity.vel)
}
}
// Add system to engine
engine.addSystem(MoveSystem)
To sum up…
The shift from an object-oriented to a data-oriented approach was a difficult architectural decision, but it was taken again due to scalability issues inherent to scenes that will become more and more populated and complex. An existent scene should run and reproduce better in terms of resources, and scenes to be created can now gain more complexity.
With minor tweaks in the syntax, the paradigm where we were standing till now will change for the sake of hardware, and more importantly, it will enable faster synchronization and overall better performance for all scenes.
As always, here are some useful links to explore, get a better understanding, and go further:
Entities and components
Data Oriented Programming (DCL)
Data Oriented Design (External)
ECS7 Template
