by 0xb8782cf628357ce2751a4ea3007934048fbaa672
Linked Draft Proposal
Should stale profiles be removed from the Content Servers? (DRAFT)
Summary
This proposal addresses the accumulation of dormant profile entities in the content servers, outlining the necessity for their removal to optimize server resources and the synchronization time for new nodes.
Abstract
Establish and implement a content server policy for the removal of stale profile entities, aiming to enhance server resource utilization and synchronization efficiency, particularly for new nodes. The focus is on profiles that have remained inactive for over a year.
Motivation
The motivation stems from the accumulation of dormant profile entities on content servers that don’t hold any value, impacting resource optimization and synchronization efficiency. The proposal seeks to address this by establishing a default behavior to permanently remove profiles inactive for more than a year. This approach aims to enhance synchronization processes, minimizing the download of unnecessary stale data during node synchronization.
Specification
A new process will be implemented on the content servers to routinely assess all existing profiles and determine whether they should be retained. Currently, some nodes host over 2 million profiles, each consisting solely of information related to avatar configuration and outfit details.
In the event of a profile entity being lost or deleted, returning users will only need to reconfigure their avatar and outfit. No tangible content or ownership data would be compromised, given that ownership information is blockchain-based. Additionally, all owned assets remain accessible, as the 3D models for each asset will persist on the content servers.
A profile entity is composed of a definition file, two snapshot PNG images (body.png and face.png), and an entry in the local database on the server. Through this cleanup process, these resources can be released, streamlining the system’s efficiency.
You can validate the content of a profile entity by executing the following HTTP request and ensuring that it does not retain any historically valuable data. The profile entity works solely as an avatar configuration, capable of being reconstructed at any given time.
curl --location 'https://peer.decentraland.org/lambdas/profiles' \ --header 'Content-Type: application/json; charset=utf-8' \ --header 'Accept: application/json' \ --data '{ "ids": [ "0xAddress" ] }'
Impacts
-
The bootstrapping time for a new Catalyst node will see a substantial improvement. Currently, the process involves handling 2 million profiles and conducting blockchain validations for each profile entity, taking approximately 36 hours to start a new node.
-
Enhance the efficiency of queries on the content server database for quicker resolution of user profiles.
-
Reduce the storage footprint per Catalyst node by releasing several gigabytes of data.
-
In the event of a profile deletion, a returning user will retain all their assets. The user will only need to navigate through the avatar configuration window, a fast and straightforward process.
Implementation Pathways
A new process will be created on the content servers to implement the stale profiles policy. This process will periodically scan all profile entities in the database, comparing their timestamps with the current date. If an entity’s timestamp is older than a year, its database entry, associated entity file, and images will be removed from the server.
The development of this process will be executed in stages:
-
Dev Stage: Initially, it will be developed and tested in a development Catalyst in zone environment using Sepolia network. Once proven effective after testing it will move to a peer-testing node to revalidate its functionality with production data.
-
Pre-Production: The peer-testing nodes contain all the production data, yet they function independently as standalone nodes. Any impact on these nodes does not propagate elsewhere. To validate the policy, it will undergo testing using this node with production data, with continuous monitoring of database changes for profile entities and tracking disk storage size. The process will be operational on a pre-production node for a minimum of one month to ensure its functionality aligns with expectations.
-
Production: To mitigate risks, prior to applying the new version, backup snapshots of the servers will be generated. This precautionary measure ensures a viable disaster recovery pathway in case unforeseen issues arise during the rollout. A new catalyst version will be created and gradually be rollout to the nodes, once its behaviour is confirmed in a couple of production nodes it will be shared with all the catalyst owners to perform the upgrade.
Conclusion
The proposed policy for the permanent removal of stale profile entities, those inactive for over a year, presents a solution to optimize server resources and improve synchronization efficiency for new Catalyst nodes. By implementing this policy, the content servers can be more efficient and minimizing the download of unnecessary, dormant data during synchronization.