ADR-123: Schema and serialization of SDK components

More details about this document
Latest published version:
https://adr.decentraland.org/adr/ADR-123
Authors:
leanmendoza
menduz
Feedback:
GitHub decentraland/adr (pull requests, new issue, open issues)
Edit this documentation:
GitHub View commits View commits on githistory.xyz

Abstract

This document presents the definition and serialization of SDK components for the Decentraland protocol. The schema and serialization to be used are Protocol Buffers and the source of truth for the components live in the decentraland/protocol repository. This standard is valid from SDK7 onwards, the former DecentralandInterface-based serialization is deprecated with a sunset date TBD.

Context, Reach & Prioritization

As of SDK6 components are serialized using JSON and JSON envelopes (JSON serialized inside JSON strings). That worked well for a small scale with few entities and simple scenes. But the performance limitations of using JSON play against the potential of the platform.

The general approach was to use JSON, now there are some components that are known for sure that need a special serialization to optimize speed: Transform is a clear example. Since transform can be mapped 1-1 with a memory struct of floats, it makes sense that the serialization of the component is only a copy of the memory of the struct.

That may not be the case for components with complex structures, like Material, which has strings on it and other resources (or sub structs) like textures. The memory mapping is non viable.

For this problem, there are some options that need to be considered and evaluated:

Solution Space Exploration

This pair, definitions and serialization, can live together from a parent definition, for example, a schema that auto generates both, the serialization and its type. A point to be remembered is each component can have its own serialization, even a JSON one.

The proposal option are:

Custom runtime schema-based serialization

Using a JSON schema that generate a versionable append only component

Primitive types

Its representation is a standard and the CPU understands it.

Compound only from primitives and fixed types

Here we can introduce the recursive condition, all primitives types are fixed in their lengths, but if we have a struct with all fixed length fields, the struct is also a fixed length one.

Other supported types (non-fixed-length)

If a component is a fixed-length struct, it can be mapped 1-1 in memory.

Approach and examples

Directory tree

The components are placed in a directory components and each component has its own directory named in pascal case e.g.BoxShape. In the component directory, there will be the version of the component:

components
├── Transform
│   └── v1.json
└── BoxShape
    ├── v1.json
    └── v2.json
Maintenance (strict similar)

The only maintenance rule is each future version has to contain the previous one. We could add a field at the end of the root struct, but we can not remove one. If we wish to support optional fields, we can create a new type that stores the state of existence of the mentioned field. Optional types break the fixed-length and non-fixed-length components cannot be optimally copied to the struct in the language that be able to do it.

We neither can modify a child struct.

What about if a fixed-length struct in v1.json mutates to a non-fixed length in v2.json? Well, the first part of v2.json is v1.json and we append fields at the end. With this rule, the first part of v2.json is a fixed-length one.

Example:

  1. We have a flag in a BoxShape called isTriggered that indicates if a BoxShape was clicked.
  2. This flag was introduced in components/BoxShape/v1.json, at the begging of its life ECS 7.0.0.
  3. In ECS 7.3.0 we want to add the feature that indicates with what button was clicked.
  4. It sounds seductive to change the isTriggered from a boolean to a struct that has the flag and the button field. This MUST NOT be allowed.
  5. Instead, we add a field at the end with isTriggeredV2 or a triggeredButton. In this case, we'll create isTriggeredV2 as a struct with two fields: flag: boolean and button: int32.

So far if the Renderer sends a BoxShapeV2 to a scene, the scene will be able to read it, no matter if the scene was compiled with ECS 7.0.0 or ECS 7.3.0. The same occurs on the opposite side if the renderer has the implementation of BoxShapeV2 as the last updated version also can get a BoxShapeV1 from the ECS 7.0.0 scene.

Code generation

Transform component:

{
  "schema": {
    "type": "struct",
    "fields": {
      "position": {
        "type": "struct",
        "fields": {
          "x": { "type": "float32" },
          "y": { "type": "float32" },
          "z": { "type": "float32" }
        }
      },
      "rotation": {
        "type": "struct",
        "fields": {
          "x": { "type": "float32" },
          "y": { "type": "float32" },
          "z": { "type": "float32" },
          "w": { "type": "float32" }
        }
      },
      "scale": {
        "type": "struct",
        "fields": {
          "x": { "type": "float32" },
          "y": { "type": "float32" },
          "z": { "type": "float32" }
        }
      }
    }
  },
  "metadata": {
    "componentId": 1
  }
}

Research of existing solutions

To give a real comparison of these approaches, they were tested two recurrent components: Material and Transform. Material is a full optional field component and Transform is a component that probably never changes and it'll be sent a lot.

The scenario was:

Results

https://docs.google.com/spreadsheets/d/1RGFkudVUMlOkNaQZFYAr0qEvVlrULmWp_F1fGBw0pwM/edit?usp=sharing

Serialization of 12,000 full filled materials Serialization of 12,000 empty materials
Type of serialization Time [ms] Size [KB]
EcsType 54.63 2784
EcsType with optional 92.17 2592
FlatBuffer 88.44 12288
Protocol Buffer 55.23 2988
Deserialization of 12,000 full-filled materials
Type of serialization Time [ms]
EcsType 150.26
EcsType with optional 134.78
FlatBuffer 185.66
Protocol Buffer 65.14
Serialization of 12,000 Transforms
Type of serialization Time [ms]
EcsType (raw) 5.57
FlatBuffer 8552.9
Protocol Buffer 99.52
Deserialization of 12,000 Transform
Type of serialization Time [ms]
EcsType (raw) 14.6
FlatBuffer 37.45
Protocol Buffer 16.49
Serialization of materials
Serialization of transforms
Serialization size of materials

Short analysis

Flatbuffer will be discarded in this analysis because it seems to be not compatible with having an external byte builder buffer, its optimization is more oriented to a C++ implementation and an ecosystem with all Flattbuffer. Another inconvenience is its code-generator for typescript, it uses classes and these would introduce extra retyping work, ECS 7.0.0 is a strictly data-oriented paradigm.

Protobuffer demonstrated a powerful performance in complex types, and raw EcsType is in the same level, but for the Transform case, raw is around 20x faster for serialization.

For the maintenance, Protobuffer has a clear advantage. Since all their fields are naturally optional, the messages can be changed completely. EcsType approach doesn't have, and for each version it'd be necessary to send the previous data.

Conclusion

RFC 2119 and RFC 8174

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 and RFC 8174.

External links

License

Copyright and related rights waived via CC0-1.0. DRAFT Living