[DAO:c849148] Use A.I. to Create Scenes in Decentraland

Governance
1

by 0x46c02de46629a2babe7c9ba1d9ebf0804d4f99aa (adi)

Should the following Tier 4: up to $60,000 USD, 6 months vesting (1 month cliff) grant in the Platform Contributor category be approved?

Abstract

DCLBuilderAI is a tool to generate scenes and other assets using natural language prompts (ex: create a scene with a park and a bench). Within seconds, it will generate the scene using the prompt as docstring, and a multi-layout view will display both the code for the scene and the preview. Users will be able to edit the code and preview it on the fly. For convenience, we’ll also add the option to connect your wallet and publish the scene to your land. We’ll be using Codex(an immensely powerful GPT-3 programming model) and Tensorflow for the project. We’ll later add support for using DALL-E in the scenes too.

Watch this video preview of one use case that we got working by feeding in some examples from the SDK: DCLBuilderAI - YouTube

Grant size

56,000 USD

Beneficiary address

0x46c02dE46629a2bAbe7c9ba1D9EBF0804d4f99AA

Email address

asaxena1415@yahoo.com

Description

Codex is the model behind GitHub copilot and a GPT-3 derivative which is itself trained on more than 170B parameters, making it ideal for code-generation. This is already starting to gain popularity in the industry (emerging projects: Deep Learning Trends: top 20 best uses of GPT-3 by OpenAI)

Decentraland can benefit a lot from leveraging these OpenAI tools. Builders will be able to save thousands of hours spent on manually creating scenes, assets, avatars, etc.

Here are some basic use cases we’re planning to integrate with our tool:

Generating Custom Entities, Components, Actions

  • Run primary queries like “create a scene with a lot of trees” and secondary queries like “add a moving bird everytime a tree is clicked”
  • Actionable scenes like “create a drunk zombie”, “add unicorns next to the zombie”, “a jumping shark”, “a jumping shark with a drunk zombie”, etc.
  • Event based assets like “party rooms”, “sports avenue”, “cube jumper”, etc.

For the scenes, there will be a split pane, with an editor, and a previewer - code generated through the prompt will be visible on the editor, and the user will be able to see its preview on the previewer, manipulate it, provide feedback, and even publish it to their land by connecting their wallet on DCLBuilderAI. We’ll be using SDK7 for enhanced performance/usability for our project.

We’ll create a separate pipeline to ingest all the available training data from decentraland scenes repo, SDK, and other public sources, and feed it into the codex da vinci model, fine-tuning for it to work with DCL (and generate valid output) will happen through OpenAI as well. On top of this, by giving the users the option to provide feedback on the accuracy of the results, we’ll be able to optimize feedback loops, decrease statistical errors like type-IIs, and just deliver better content. The model, training, fine-tuning, and api will be part of codex and will come from OpenAI whereas Tensorflow will be used for the data ingestion pipeline process.

Note that this tool will not be assisting with minting and listing assets on the DCL marketplace but will be providing an editor and preview option for working with assets. Besides, the results are not guaranteed to be 100% accurate and will hopefully improve over time with the help of the feedback loop.

Specification

The project will require working with a number of key services such as OpenAI, Tensorflow, MetaMask, DCL SDK7, and other web development frameworks/tools. The tool will be used through a web app developed using React for frontend and Node for backend. Setting up and maintaining the infra will cost ~$12K a year.

Using OpenAI for training/usage would be expensive as well - we plan on using the da vinci language model – one of the most powerful ai models – to deliver state of the art results for our customers.

OpenAI cost for running queries: $0.0200 / 1K tokens (which is roughly ~750 words)

In order to make the tool more accessible to the community, we’ll use up to $8k in funding to fund its usage so that users in the DCL community can play around with it for free. Once we’ve exhausted this funding, we will directly charge the end users to foot in the costs for their use cases.

The rest of the funds will be used as dev salary for our team - we all have skillsets in NLP prompt engineering/model training/data pipelines but would like to bring on board one data scientist/engineer too to help speed up the project development.

The project will be open-source and we’ll also try to partner with existing projects to help with components like publishing to the lands or making the previewer generate the scene as changes are made to the editor on the fly. We will later also work toward adding support for using DALL-E with GPT-3 to generate 3D models from text.

Breakdown of Costs:

Work Cost
OpenAI Project Community Access to DCL $8K
Web Tooling, Infra for frontend and backend, maintenance of project $12k
Total Salary $32k
Misc. Expenses $4k

Personnel

Aditya - Software Engineer

Tave - Software Engineer

We’re both full-stack software engineers with prior experiences at companies like Amazon/Microsoft with exposure to large-scale AI/ML-based projects, and are also the team behind a peer-to-peer rental marketplace project called RentParcel delivered through a successful grant (Democratize Access to Decentraland for Renting, Marketing, and Hosting Events). We’ve been involved with several DCL landowners in the last 1 year, generating value for them by securing land rentals.

Roadmap and milestones

We plan to complete the project within six months, here are the goals we’re setting:

  • Months 1-2: Work on developing the data ingestion pipeline, model training, prompt engineering, fine-tuning, creating backend for accessing OpenAI APIs for Codex.
  • Month 3: Work on the UI for the web app that will use the tool, create scenes, other assets, previewer, and publish option to the lands.
  • Month 4: Work on integrating the backend with the frontend.
  • Months 5-6: Work on adding option to connect wallet, publish to land, and beta-testing the project, and granting access to the community.

Vote on this proposal on the Decentraland DAO

View this proposal on Snapshot

2

Voting Yes!

I think is great that the community gets to integrate cutting edge AI technology into the creative pipeline. Large Language Models are very good at writing code (I’ve been using GitHub Copilot), so I think what is proposed here is doable.

3

Vote: Yes

This tool appears to extend the capabilities of DCL’s current state. It could improve access to users who might use text to speech due to limited mobility. This is a case that takes advantage of this platforms open source nature. I’d love to see more proposals like this one.

4

Vote: Yes

This is a fantastic proposal. It could help get more scenes up & running on lots of currently empty LANDs. I agree with AwedJob, this extends DCL capabilities, even has the potential to be fully integrated. I would be a heavy user of this.

5

In order to make the tool more accessible to the community, we’ll use up to $8k in funding to fund its usage so that users in the DCL community can play around with it for free. Once we’ve exhausted this funding, we will directly charge the end users to foot in the costs for their use cases.

Will there be some user lock here so the initial budget can’t all be used up by trolls? are generated scenes just used one time or will the code generated be added to a scene pool / github repo that others can see / use?

We plan to complete the project within six months

But the infracosts are for 12 months - so that means you won’t ask for another grant for at least a year, and maybe not at all as this project is meant to start generating income. is that right?

6

Hey @dax, our plan is to make the first few releases under a private beta, knowing beforehand about the user and their use case. If we notice any form of abuse, we’ll definitely cut off their access. We’d definitely like to cache some of the scenes and add them to our public repo for others to benefit too, it’ll be contingent on user’s consent :slight_smile:

And yes that’s right, the plan is to make the project financially self-sustaining so that it can cover any operational costs without us having to ask for another grant :pray:t2:

1 Like
7

Really cool concept I can’t wait to try out!! Thanks for building this!

8

ok i think YES from me. but please explain to me like an idiot cuz i am lol. we type in what we want our scene to look like in decentraland and this Ai generates it for us and we can deploy? if so sounds epic thanks.

9

Hey @AaronLeupp, your understanding of the general idea is correct, that the AI would be able to spit out scenes based on what you type and you’d be able to play around with, and publish them! Basic existing objects/assets, manipulating game mechanics, and other simple dev work will not require any optimization beyond what the AI generates. In the beginning, however, especially for complex scenes, it will not be perfect and some edits might be required but over time it will train itself to reduce the chances of failure with feedback from users. It’ll be a utility on top of the builder and will be having a huge impact on anyone building in dcl saving them a ton of time. There is plenty of training data available already with the dcl scenes repo and other open-source projects, so we’ll train our model to use it conveniently through natural language prompts.

One thing to clarify is that we’re not proposing building 3D models from text for this grant, current AI models cannot give the desired results for this - in the future though, we can use DALL-E and GPT-3 to create textures to apply to 3d models at least!

10

I really like the proposal and I believe it could lower the access barrier to Decentraland for creators without technical skills. Are you thinking of training the model just with Decentraland scenes or are you planning to use 3d model datasets already available? Asking because there is almost no metadata attached to the DCL Scenes so it might be slow at the beginning to train the engine.

The only thing that worries me a bit is:

Once we’ve exhausted this funding, we will directly charge the end users to foot in the costs for their use cases.

Who are we? If you are gonna be adding a surcharge to the OpenAI price it means that the DAO would be seed investing in a business. If that’s the case, I’d like to see some kind of return to the DAO Treasury. I’ll defer my vote until this topic gets clarified :slight_smile:

1 Like
11

Hey @ginoct, so nice to see you here :slight_smile: - By we, I meant, we ‘the team’ behind the product, and we will not be adding in any extra surcharge to the user - keeping it free and the costs transparent to the community. By footing in the costs, we meant to say that we would have the customers directly cover the OpenAI costs once we exhaust the $8k funds for providing free access to the tool. If customers can directly cover the OpenAI operational costs, we won’t be having any losses and will not need to worry about making another grant too. We can have a page on our website too that lets users directly compare costs with OpenAI so that they are sure of our genuineness.

For your other question - we will largely use the publicly available scenes, and yeah we’ll do data pipeline transformation to create the metadata to train the model, as you said, the results may be not as great in the beginning, but once enough time passes with the feedback loop, and the model learning on its own - the quality would drastically improve :slight_smile:

12

I think this is a great idea, so I’m voting yes, but I’m a bit worried about the timing, because of ECS7.

Despite the fact that ECS7 will have a compatibility layer, for this project to be valuable long-term for Decentraland I’d expect it to be built on top of ECS7 and not ECS6. This is why I think you should wait until ECS7 is released to develop this.

13

We’ll make the switch without requesting any additional funds for it :slight_smile: Thanks for the support!

14

It’s one of those things that sounds cool but we all know that a few people will use it. Why? Because this won’t work to do complex things, and for doing simple stuff, we already have the builder.

In addition to this, Decentraland Foundation plans to mix the builder with the SDK at some point. I guess that will naturally lead to improving the builder. Maybe in the future, we will have drag-and-drop behaviours.

15

I have actually already received several messages from builders/architects about feedback and ideas to add to this proposed product - so I really think if the proposal can generate this level of interest, a lot of people will be using the actual product :slight_smile:

Sorry again but I am not sure what you mean by simple things and complex things for this? Codex is powerful enough to be able to create complex apps (front end + back end) so I was just curious why you were thinking it will not be able to get fine-tuned to work with DCL.

This project is about adding OpenAI tooling to help with creating scenes, so again, I’m not really sure why mixing the builder with SDK will accomplish what we’re proposing :slight_smile:

I respect your opinion but I do think that the comment is unfair and if this assessment is what has prompted the vote - I do urge you to take one more look :pray:t2: Thank you!

17

Voting YES. I believe that scene creation with the help of AI is going to be an important part of the creator journey in the future. This proposal is a good first step in that direction and I’m excited to see the results and what’s delivered!

18

Use A.I. to Create Scenes in Decentraland

This proposal is now in status: PASSED.

Voting Results:

  • Yes 87% 3,375,150 VP (97 votes)
  • No 13% 533,914 VP (8 votes)
19

Use A.I. to Create Scenes in Decentraland

This proposal has been ENACTED by a DAO Committee Member (0xbef99f5f55cf7cdb3a70998c57061b7e1386a9b0)

Vesting Contract Address: 0x087b2D11c4410013e41E428a3Be08e0052096617

20

Dear community,
The Grant Support Squad was created to support grantees to achieve their goals and to take care of the DAO Treasury, with the main goal of guaranteeing the effectiveness of the Grants Program.
We had a meeting with the grantee and we checked that this project has already started and will continue during all of 2023. However, the vesting contract created is not following the timeline of the roadmap proposed.
For this reason, we suggest that the DAO Committee (@yemel @HPrivakos) create a new vesting contract with a duration that coincides with the schedule presented (that will be published by the Grantee). In this sense, the new vesting contract should start as soon as possible with a duration of 8 months, preferably without a cliff.
Best regards,
The Grant Support Squad

1 Like
21

Here’s our updated roadmap - we’re mainly extending it in order to accommodate for ECS7 in the model:

Month 2 End:
- Have a basic front-end ready for the website (It will not be fully functional but we would have a basic design ready that shows an editor to work with the code and the prompt, preview scenes, connect your wallet, and publish the scenes. The website will be up and the code published to our GitHub.
- Backend infra set up to be completed, the data ingestion pipeline to support the reinforced learning through the feedback loop would be set up, tested, and ready to be used.

Month 3 End:
- Hire Data Scientist to help with the model training and assist with improving the AI for the scene generation. The model is being fine-tuned on SDK7 data so that it is compatible with ECS7. By this month’s end, we should have a CSV data set with 1k examples tested and ready that are compatible with SDK7. The data set will be shared on our GitHub.
- Have functions implemented for code generation on our website (using the 1k examples data set we would have prepared).

Month 4 End:
- Open the website up for access to users. We’ll figure out the best way to open it up so that the $8k we have for this is utilized in a manner so that it can used by the maximum number of DCL users.
- Augment the model size to 3k examples compatible with SDK 7. Both code and the data set will be accessible on our GitHub.
- Connect the tool to the pipeline so that the data set now augments with the help of the beta testers and not just us.

Month 5 End:
- Make the functions to preview the build functional on our website. As well as offer suggestions to users on the entities or functions that they can add to their scene that they would be developing using the BuilderAI.
- Augment training set to >5k examples.
- Integrate with a stable diffusion AI to let users generate textures too for their scenes or other assets.

Month 6 End:
- Create a library of scripts from the training data on our website that doesn’t require paid access.
- Have a 10k+ training set ready.
- Make connect your wallet functional.

Month 7 End:
- Make the function to publish build on the land functional.
- Continue working on augmenting the training data set with SDK7.

Month 8 End:
- Make the website fully functional, user should be able to create scenes using AI, preview them, download them, and publish them to their land.
- Continue working on augmenting the training data set with SDK7.

Month 9 End:
- Have a 50k+ training data set ready for DCL SDK7. The model should now augment on its own using the pipeline and the feedback loop.
- Open the tool up for broader access to anyone interested in Decentraland.

2 Likes