[DAO:6c6bcac] AI dapp to transform any 2D PFP NFT into a 3D Decentraland Wearable/Avatar

Governance
1

by 0x51159c749702295e745319935e4eddde238f3a7f

Should the following $108,000 grant in the Platform category be approved?

Abstract

We are excited to present an AI-based solution to an unsolved metaverse problem and lowers barriers to onboard NFT collectors to decentraland:

Problem:
Thousands of NFT collectors want to use the PFP NFTs in their wallets as avatars across platforms. Especially in 3D metaverse environments, this is difficult to pull off: The costs of rigging and format conversions of 2D PFPs to 3D Avatars are high.

Solution:
Beamit.space combines several cutting edge AI models to transform 2D PFPs into 3D Metaverse Avatars for Decentraland. Any PFP you want - you can soon use it in the metaverse. And this can be done by anyone:

1000x faster than ever before.
100x cheaper than ever possible.

Grant size

108,000 USD in DAI

Project duration

8 months

Beneficiary address

0x51159c749702295E745319935E4EdddE238F3A7F

Email address

decentraland@beamit.space

Description

We believe Beamit.space is a game changer in 3D Avatar generation and will change the game for Metaverse onboarding. Our early prototype proves that we can make this a reality within 8 months.

The app solves this in 3 stages:

Beamit.space is using cutting edge technology. We are merging AI with 3D rendering and web3 technology in a unique way.

1. Generative AI

Our Generative AI (gAI) module enables users to complete PFPs to create a full body image. Our prototype (using Dall-E) shows, that decent results are achievable without even explicitly training the model. So we anticipate substantial improvement potential from training our own model (we are currently favoring training the open source model Stable Diffusion). The goal is to generate a highly accurate 360° full body image of the PFP (frontside and backside, eventually also side views).

External Image

2. Depth Prediction model We use a cutting edge Depth Prediction model to transform the frontside and backside 2D full body images into 3D models. Even though the model was trained on photography, the results for illustrations and even pixels design are stunning. We anticipate to be drastically improving the prototype results by training the model on appropriate datasets.

External Image

3. Blender Cloud (a. Background removal, b. Merge & c. Rigging)

The third step serves to set to finetune the raw renderings, merge them to a full 3D Avatar and rig them with the Decentraland skeleton rig. For this step we are using the open source 3D creation suite Blender.

a. Python scripts first will detect and remove the background from both the frontside and backside gAI results.
External Image

b. Next the backside and frontside view 3D models are combined to create the initial stage of the full avatar 3D model.

c. The automated rigging is the most challenging part of the project. We are working on our own AI model based on existing 3D Shape Recognition models, as well as prediction models on how to position the skeleton in the avatar mesh. However, as perfect results are not to be expected in the ealry stage of this project, we will provide a blender cloud browser-application the user can use to adjust the rigging manually, to get a functioning avatar ready for publishing/submission.

External Image

Results: All of the blender add-on scripts as well as the custom trained AI models will be open sourced.

You can now check our prototype (with Steps 1. and 2. in place) and test it with your NFTs - please request via website form beamit.space.

Roadmap and milestones

We anticipate the project to be ready within 8 months from now - with the result of an open sourced alpha release as result. We are working according to this roadmap, and will update here as well as in detail on our medium:

8 - 2023 Proof of Concept 3D frontside body, Ethereum
Status of the project to date.

9 - 2023 Frond side + added backside Render generative AI model (gAI )

  • gAI model producing Both front & back side results.
  • blender cloud set up and script testing.
  • Project presentation & start of social media activities.

10 – 2023 Full Body 3D Render AI model (zoeAI)

  • custom 3D rendering model based on ZoeDepth depth prediction model (for front and back).
  • Community building activities.
  • Unisat Wallet Ordinals integration

11 – 2023 Refined gAI model

  • Refined custom gAI model to achieve optimized results by training the model on web3 PFP data.

12 – 2023 – Blender Cloud: remove background

  • Remove the backgrounds of all 3D body perspectives via python script.

01 - 2023 – Blender Cloud full body merge

  • Creating full body 3D Avatar.

02 – 2024 refined zoeAI + rAI model

  • Trained models to further increase render quality
  • Community building: private Alpha whitelist.

03 - 2024 – Blender Cloud: Rigging

  • Rigging the 3D Avatar in the cloud, with basic script support.
  • Community building
  • private Alpha

Vote on this proposal on the Decentraland DAO

View this proposal on Snapshot

2

Hi everyone - sorry the images didn’t load. please check the prototype screenshots here:

[g AI samples 1 — Postimages](https://beamit generative AI model (prototype screenshot))
[zoe AI samples — Postimages](https://Depth prediction model 2D-3D transform, prototype screenshot)
[blender5 — Postimages](https://rigging the avatar with decentraland skeleton)

lmk about your questions & request prototype access

4

Hey, first of all great job on addressing this problem. I’m a maxi on making 3D modelling and design more accessible and an app that could deliver on this would be a game changer for the space.

Unfortunately, for me I’d need to see more of a proof of concept that your team can successfully create a low-poly, rigged, weight-painted avatar before voting yes. I imagine scaling and maintaining for DCL would be a huge feat and that a budget of this type would be more suitable to adapt your existing technology to the platform.

All the best.

5

Thanks for your reply @SugarClub. We would very much appreciate it if you take this project seriously just as we do, already having put a lot of work and money into it. Sure, we are at an early stage aiming to do something which has never been done before. Maybe too early for requesting a grant here.

You can find info in the proposal, that fine tuning the models requires a lot of processing power and dedicated work - and this is why we request funding. Sure, we can and will provide a 3D model of how we aim to achieve automated results, as well as the methodology of production.

Thank you for the feedback!

6

thanks @amanda_mod - the prototype with the first 2 ai modules is online and available on request. Surely, we can def. provide a more advanced proof of concept. working on it!

8

While i love the premise, I question how this benefits Decentrland specifically? It sounds like you have planned to build this regardless and yes we have a grants program but it is for those projects that benefit DCL. This is a project for way more than DCL and does nothing to even promise that by using this people will come into world. Have to vote no.

9

PFP Avatar sounds exciting, even more so since I have the name :rofl:

10

hi, thanks for the feedback. to respond to your remarks:

“Unfortunately at this stage I cannot support this because you don’t have a prototype that shows a qualifying 3D model that would fit a gaming environment.”

  • we can of course show a prototype and will do so in a couple of days. however, it is for developing the automation process we are requesting funds for, and building this takes time and processing power for testing, which is why we need funding.

“Furthermore the tools of depth extraction that you mention are already existing and free to use online for 2D to 3D . Similarly a lot of AI apps that can be used to create 2D images without any financial costs such as stable diffusion.”

  • Yes these models are available, but none of these offer ready to use results of a 360 3D avatar directly from an NFT based on the combination of 3 different models. Also we explan in the proposal that we want to train the models on datasets that help get results especially for our use case.

“AI is a trendy topic but is also a very sensitive subject regarding the data on which the models are trained and the methodology of how it is being used to generate models. Will you be transparent on the source of the data you have trained ?”

  • We will open source our models and also reference the datasets.

“Will you give free access to those tools you are developing ? Will it be opensource ? Will it have a
Subscription model or paid features ?”

  • Yes we will open source everything as stated in the submission. However, ai-processing and 3D rendering requires computing power that needs to be covered by the user. Anyone could run these models, but we also plan to offer a service that will require users to pay for the rendering. We could also agree on a shared revenue model with the DAO if there is interest in that - then I would prob. better submit this for the accelerator?

“FYI Asking for funding for a project that will have a paywall to access to ( with tools that are already available online for free ) will meet a lot of resistance from the DAO .”

  • The funding is requested for a tool integrating AI-generated 3D Avatars specifically for DCL. Processing will cost money though - we assume the cost per Avatar (processing only) will effectively be very low compared to manual generation. I see your point here, so we are happy to offer the dapp to DCL users for just the processing cost if we receive a grant.
11

First of all we want to build this for DCL as it is the pioneer of a decentralized metaverse.

Second, DCL Avatars have very specific requirements. A big part of the results of this project can not just be used for a different metaverse.

Third, is that we plan to integrate NFTs from multiple blockchains. Users will be incentivized to create a DCL avatar from their NFTs regardless on which blockchain they hold them. And of these new users will likely discover and explore DCL (ofc I cant promise that, but even in-world games projects can’t promise that).

You have a point that we want to build this project regardless of the outcome for the grant - because the utility is obvious. We also considering other metaverses. There are very many being built right now, and an app like ours has a great use in this “metaverse of metaverses”. We want to contribute to making transitions as easy as possible, lowering the barriers between these worlds.

I believe users will enjoy having the freedom of easily switching between these worlds, as they already do. And that metaverses with lower entry barriers will have a competitive advantage over the ones trying to isolate.

1 Like
12

Voting No, my understanding is that the technology is not ready to transpose a 2D image into a 3D model with high enough fidelity and accuracy to turn PFPs into Wearables.

What is possible now:

  • Take multiple photos of a item and turn all those photos into a 3D model. The 3D model may need fixing.
  • Take an image of a human face, and turn it into a 3D face.
  • Describe a very simple object (like a chair) and get a nice low poly 3D model.
  • Take an image and compute the depth of each element (like in you example). That is technically 3D, but not the kind of 3D needed to make a wearable.

I would need proof that there is a robust technique that can take a PFP image, extrapolate the missing information, and build a polygonal 3D model suitable to rig and turn into a wearable.

13

yes, not ready yet, but the ingredients are here. that is why we are asking for a grant, to build this. I see we didn’t describe our approach enough, and also prototype of a rigged model was requested - we are working on that and will present here soon.

however, if you want to see the 100% ready product before voting yes, then we will not need funding anymore - we need it to build. :man_shrugging:

14

While not entirely the same, I have access to an AI 3D app. It is only for doing furniture at the moment but is rather impressive. I know there is another that is for shoes currently. So I do believe the ability is not far off.

IMG_0594.PNG
IMG_0594.PNG1284Ă—2778 97.9 KB

Ive been playing with a few different things.

1 Like
15

AI dapp to transform any 2D PFP NFT into a 3D Decentraland Wearable/Avatar

This proposal is now in status: REJECTED.

Voting Results:

  • Yes 1% 6 VP (15 votes)
  • No 98% 8,931,817 VP (69 votes)
  • Abstain 1% 859 VP (2 votes)