by 0xedf458ce355c755e1973778b51b917066d4c0bad (8MetaGames)
Should the following $24,000 grant in the Platform category be approved?
Abstract
Open Source tools for integrating LLMs (AI) into Decentraland.
This grant proposes integrating Large Language Models (LLMs) into Decentraland, offering creators and developers out-of-the-box access to AI technology. Aimed at enhancing user experience and fostering innovation, it will provide open-source tools for building advanced, AI-powered virtual environments, marking a significant step in leveraging AI breakthroughs within Decentraland.
Customizable AI Characters: Front-end interface for creating AI characters.
Multilingual Interaction: Characters respond in users’ native languages.
Backend Setup: Connection to various LLMs including OpenAI, Authrophic, Azure, and local installations.
2. Frameworks and Tools for LLMs
MemGPT: Framework for crafting detailed AI characters.
AutoGen: Tool for generating interactive NPC groups.
Ollama: Facilitates local LLM integration.
Prompt Journey: System for mapping NPC interactions.
Integration Instructions: Detailed guidelines for integrating OpenAI and local LLMs, such as Llama, Mistral, and SDXL models.
3. Stable Diffusion in Decentraland
Text-Prompt-Based Image Generation: Backend integration for dynamic visual displays.
Use-Cases
Virtual Guides and Assistants: AI characters helping users navigate Decentraland, providing event info, or assisting in tasks.
Interactive Storytelling and Roleplay: Customize characters for interactive storytelling and roleplay with users.
Virtual Companions: AI characters as companions for conversation and social interaction.
Educational Tools: AI characters in interactive learning modules or virtual classrooms for various subjects.
Customer Service Representatives: AI characters in virtual stores or service centers for user assistance.
Existing OpenAI Integration
We’ve initiated AI technology integration in Decentraland with OpenAI for image generation, showcasing our commitment to advanced AI in virtual environments.
The Foundation is already using ChatGPT in one of the NPC i believe, you don’t need a 24k usd grant to do a command to a server and write a few prompts
While it’s true that the Foundation has utilized ChatGPT for NPC interactions, our project aims to go beyond basic command-response functionality. Our proposal involves:
Advanced Integration: We’re not just implementing simple commands; we’re integrating a more sophisticated range of LLMs, including local installations and other models like Claude, which offer diverse capabilities.
Multilingual Support: We plan to implement multilingual interaction capabilities, making the NPCs more accessible to a global audience.
Comprehensive Frameworks and Tools: Beyond basic chat functions, we’re developing unique frameworks like MemGPT and AutoGen, along with tools for more complex NPC interactions and storyline integrations. Community Involvement and Open Source Contribution: A significant part of the grant will be used to create detailed documentation and support community contributions, fostering a collaborative environment for continuous improvement.
@HPrivakos, I think @Tony is being very fair with the price, possibly you have good skills that allow you to do things like this in less time, but the truth is that there are no applicants who propose this for Decentraland, and if we look at platforms like Fiverr, the prices to implement all this are currently quite high.
I believe that all tools with AI should be implemented as soon as possible, not only because of the numerous functionalities that it can provide to the platform but also to adapt Decentraland to these new technologies.
We have already seen in the past many virtual worlds that were left behind in the web3 race, I think the same can happen with web3 virtual worlds that do not implement AI.
i voted no , I’m not saying your projects isn’t good, but I have mentioned multiple times that the platform itself has many issues that need to be addressed. I hope that login, user experience, graphics quality, and scene download speed, among other issues, can be prioritized and resolved for first place.
if you need to use a lot of external services, is all their code open source? what happens if one of them goes under? and how will you handle costs in the long-term?
Thanks for your feedback @LandlordDao . I know the platform has issues – I’ve been working with it for two years. Our team talks with the SDK and Explorer teams to solve problems as they come up. But the big fixes for the platform are really for the Foundation to handle, not the DAO.
Still, I think it’s important to keep bringing new ideas while the Foundation improves the platform. Innovation happens in many ways, not just one.
Absolutely, all our code will be fully open-source under the Apache 2.0 License. The goal of this grant is to develop tools that enable creators to interact with LLMs easily.
Regarding external services and their costs, I realize my initial description might have been unclear. We don’t necessarily rely on a lot of external services. However, we do offer options:
Access to proprietary services like ChatGPT and Claude is available for those who prefer not using their infrastructure and would rather use an API.
Importantly, we’re focused on developing everything needed to run LLMs locally, whether on consumer-grade hardware or cloud services.
We won’t be handling any costs directly. Our aim is to provide tools that allow creators to use LLMs within Decentraland. Choosing which service or provider to use will be up to the user.
For example, we’ve developed a simple Chat-Bot for Decentral Games. It’s fully local, uses open-source LLMs and libraries, and only requires a PDF or a webpage link with project information.
Check it here: https://twitter.com/TonyMo74/status/1745380483389956509
We’re collaborating with Pablo from GolfCraft to bring similar functionality to their game as well.
On a separate note, I’m aware that our grant competes with your DCL Metrics grant. I’ve noticed a few ‘no’ or ‘abstain’ votes on ours, possibly due to budget concerns if both are approved. I have great respect for your work in DCL Metrics, which is an essential resource. You definitely deserve the grant. I plan to resubmit mine with additional examples, details, and clarifications to address any concerns.
We won’t be handling any costs directly. Our aim is to provide tools that allow creators to use LLMs within Decentraland. Choosing which service or provider to use will be up to the user.
a-ha, okay i understand now, that actually is a really nice model then.
On a separate note, I’m aware that our grant competes with your DCL Metrics grant. I’ve noticed a few ‘no’ or ‘abstain’ votes on ours, possibly due to budget concerns if both are approved. I have great respect for your work in DCL Metrics, which is an essential resource. You definitely deserve the grant. I plan to resubmit mine with additional examples, details, and clarifications to address any concerns.
please let me know when you resubmit, you’ll have my support
