[DAO:f162dd3] Should we remove redundant endpoints from Catalyst? ll

Governance
1

by 0x6438c3b1fa97ba144ea38fcbcee5f0ccf4539b1d (ph33bs)

Introduction

The Lambdas service from Catalyst was originally designed to support various independent operations required by the reference implementation client. However, as time progressed, newer and more effective endpoints with identical functions were introduced, yet the service retained all previous iterations.

This is the second of a serie of proposals that aim to enhance the resilience and efficiency of Catalyst’s Lambdas service by optimizing its endpoints and removing old implementations that didn’t take performance into consideration, making inefficient use of resources. You may find the first proposal of this serie here.

The main objective of this proposal is to decrease the overhead of the Lambdas service and improve Catalyst response times by removing the redundant endpoints GET /lambdas/contentv2/contents/:cid, GET /lambdas/contentv2/scenes, and GET /lambdas/contentv2/parcel_info.

Proposal

GET /lambdas/contentv2/contents/:cid

This endpoint is currently acting as a proxy, forwarding the initial request to the GET /contents/:cid endpoint of the Content service and returning its response without any additional formatting. Going forward, clients should access the GET /content/contents/:cid endpoint from content-server directly.

GET /lambdas/contentv2/scenes

This endpoint is currently acting as a proxy, forwarding the initial request to the POST /content/entities/active endpoint of the content-server and returning its response without any additional formatting. Going forward, clients should access the POST /content/entities/active endpoint from content-server directly.

GET /lambdas/contentv2/parcel_info

This endpoint is also acting as a proxy, forwarding the initial request to the POST /content/entities/active endpoint of the content-server and returning its response without any additional formatting. Going forward, clients should access the POST /content/entities/active endpoint from content-server directly.

Provided that this proposal gets general acceptance level, the specified endpoints will be removed one month subsequent to its approval.

  • Delete the endpoints
  • Keep the endpoints
  • Invalid question/options

Vote on this proposal on the Decentraland DAO

View this proposal on Snapshot

2

About the author
I’m Alejo Ortega, a back-end developer contributing to the Decentraland Foundation in the area of Catalyst Development. This proposal was co-authored with Matias Penthreath and Hugo Arregui.

1 Like
3

Hi, thanks for proposal. Are you planning to move your previous poll to Draft stage ?

4

Hello @web3nit,

As the changes we are making to the Catalyst API are aimed at enhancing functionality rather than causing any loss, we are just seeking a general level of acceptance instead of going through a 3 stages poll.

If anyone is currently utilizing these endpoints or wish to retain them, we would appreciate gaining a deeper understanding of their use cases. This will enable us to determine whether it is necessary to maintain the functionality by potentially re-implementing them with a focus on performance, or if the proposed endpoints already provide solutions for the identified use cases.

Thank you so much for your interest on this serie, it is highly appreciated. Please let us know if you have any further questions or concerns.

1 Like
5

Thanks, all is clear. :handshake:

6

To be honest, I’m worried about the change of GET to POST of the last two endpoints. That prevents any chance of caching those results. How much are these endpoints used, and how many of those requests are being cached? Going forward, I’d love to try to design the catalyst APIs so that they are as cache-friendly as possible. I understand it’s pretty hard, but for example, GET /lambdas/contentv2/scenes?x1&x2&y1&y2 could be replaced with some obscene amount of parallel HTTP requests that could be cached/pipelined with some redirects instead of using POST requests (maybe I’m too optimistic here).

7

Hello @esteban,

Thank you for raising this interesting topic. We agree that it is crucial to make the Catalyst API cache-friendly, and we are actively working to achieve this goal. However, in certain cases, relying solely on cloud caches may not be practical. For example, the content served by the GET endpoints /lambdas/contentv2/scenes is highly mutable, and the content pointed to by the coordinates usually changes. In such cases, caching these request would be ineffective. Instead, POST /active/entities can be used to request the same amount of information at once by taking a list of pointers, and the client could calculate the pointers in the square, which, if it is a much-needed functionality it can be implemented on the Catalyst client.

Regarding the GET /lambdas/contentv2/parcel_info endpoint, the client could use GET /content/contents/:cid if it is interested in retrieving a single entity, which would be automatically cached by CloudFlare.
Although we don’t have a CloudFlare cache for this parcel_info endpoint (it only relies on an internal cache), it’s unlikely that caching many entities at once would be helpful, as the existing CloudFlare cache has a 100% miss rate since it’s rare to request the same entities in the same order repeatedly.

Please let us know if you have further concerns about this and thanks again.

8

I’d love to delve into this a bit further with you.

When we talk about the content served by the GET endpoints /lambdas/contentv2/scenes, you mentioned that it’s “highly mutable.” This basically means that the content can change quite frequently. But to break it down, there are two types of mutability at play here: intrinsic and accidental (to paraphrase Fred Brooks).

The intrinsic mutability comes from the changing nature of the scene deployments themselves - it’s just part of the game. Accidental mutability, on the other hand, is a result of how the endpoint is consulted, with different clients asking for different X, Y ranges. So, if I’m hanging out in parcel 0,0 and you’re in parcel 0,1, we might be asking for slightly different information even though the answers might be largely the same.

Now, thinking about scenarios where multiple requests are made with the same parameters, like when spawning on the genesis plaza or teleporting to a popular area. In these cases, we could potentially reduce the accidental mutability factor. But to do this effectively, we would need a more nuanced approach to caching, something that might not be fully optimized in the Catalyst client right now.

It would definitely be worth considering some empirical data to help us understand the trade-offs better. Metrics such as response times, cache hits and misses, network and database bandwidth saved could provide some valuable insights. This could help us determine if the added complexity of this caching mechanism is truly worth it.

The ultimate goal of /lambdas/contentv2/parcel_info is to streamline things so that we only need one request to fetch the parcel deployment CID, the scene.json CID, and its contents, specifically the main script’s CID. This could be particularly beneficial when latency to the server is high (around ~200ms), as it would likely save us from unnecessary round-trips.

  • The resolution of (x,y) to fetch the parcel deployment CID
  • The resolution of the parcel deployment CID to fetch the scene.json CID
  • The resolution of the scene.json CID to its contents, particularly, the main script’s CID

I’m mainly curious about why the proposed solution involves a POST request, which as you pointed out, can’t be cached. Could we not explore a method that allows caching via a GET request? I’d love to hear your thoughts on this.

1 Like
9

Should we remove redundant endpoints from Catalyst? ll

This proposal is now in status: FINISHED.

Voting Results:

  • Delete the endpoints 100% 2,174,308 VP (57 votes)
  • Keep the endpoints 0% 0 VP (0 votes)
  • Invalid question/options 0% 0 VP (0 votes)
10

Does changing the endpoint GET /lambdas/contentv2/scenes affect our ability to query historical scene data?

11

Should we remove redundant endpoints from Catalyst? ll

This proposal has been PASSED by a DAO Committee Member (0xbef99f5f55cf7cdb3a70998c57061b7e1386a9b0)

12

i’m using /content/entities/scene?pointer=x,y is that still good to use?

13

Hello @MorrisMustang,

Currently, GET /lambdas/contentv2/scenes is acting as a proxy of POST /content/active/entities. This means, that you will be able to get the same data by consuming POST /content/active/entities from now on.

14

Hey @dax,

No changes were introduced on the /content/entities/scene? endpoint on the scope of this serie.