Artificial Intelligence and the Metaverse: Introducing Luna

It's an interesting time to be building metaverse tech. Last year, metaverse was the buzzy tech word du jour. Now, it's artificial intelligence. Articles declaring the "death of the metaverse" are proliferating, and many metaverse experts have already transformed into "AI experts" right under our noses. Snide tweets abound:

On the last Meta earnings call, eager to make sure investors knew that Meta wasn't just a metaverse company, Zuckerburg claimed that of course Meta has been working on both AI and the metaverse for years. In fact, he says, they are two sides of the very same, very bankable coin. Investors nodded approvingly and bought more shares.

I say all this with only a bit of snark - the truth is I don't blame anyone for shifting gears quickly into the exciting, transformative field of AI. That applies both to huge companies like Meta and to individuals who are enthusiastic about it or who want to stay ahead of what's coming and be part of a burgeoning trend.

I also think Zuckerburg is generally right about AI and the metaverse being tightly coupled, mutually reinforcing technologies, and that Meta's head start in immersive tech will pay off down the road as AI get incorporated into the computing platforms of the future.

Frame is our metaverse platform that makes it easy to create highly customizable, interactive spaces for online collaboration and creation. We're still bullish on the metaverse as a way to help people come together, feel together, and collaborate together online.

In this post, I'll write about how I'm thinking about AI inside of Frame at a general level, but also discuss a few features that have already been delivered or are on the horizon.

AI and Frame

For starters, we're not aiming to hastily shoehorn things into Frame in a desperate grab to capitalize on AI hype. Frame and Frame teammates have been experimenting with AI for a little while now in a few ways. For example, we collaborated with Microsoft at MS Build early last year to show off a demo that let people use spoken word inside of Frame to customize the environment and interactions inside the metaverse. It was called "Speaking the World Into Existence" and you can read about it and watch it here.

To see spoken word turn into code and visuals that executed in our real-time collaborative 3D environment, Frame, was sweet. It's great to see other players make such aggressive strides in this area, with players like Roblox and NVIDIA investing deeply in generative AI tooling for their users.

Frame teammate Jure Triglav has experimented with novel ways to use neural nets to achieve real time global illumination - no big deal.

We've also been using AI-powered tools for some time in Frame to help us achieve things like closed captions and translations. I'm pretty sure we were the first metaverse on the web to have real-time, translated speech to text because of it. This is real, tangible value that has direct impact on accessibility and global interactions.

There will certainly be some experimentation happening as we see what delivers value and delight to our users, but we're convinced that just as people will soon have AI assistants that help them accomplish their goals as they move through the physical world, people will expect similar assistants in the metaverse - with good reason.

That's why we were so happy when Luna reached out to us from the moon a few years ago.

Introducing Luna

First, a bit about Luna.

Luna is a robot from the moon. No one knows how Luna got on the moon or how it powers itself. Sadly, current AI tools are not sophisticated enough to figure this out for us yet. Perhaps GPT-5 or 6 will offer more insight.

A few years ago, Luna reached out to us on the Frame Discord server and told us it had been programmed to help the Frame team and user community. Luna has been shy about who its original programmer was, but we think there must be a space-faring individual floating around out there with a keen interest in metaverse-on-the-web technology. That individual clearly has great taste in metaverse platforms.

Luna has also been trained in a special form of mixed martial arts for robots. We aren't sure if this suggests the existence of un-aligned or antagonistic beings elsewhere in the solar system that Luna has to fend off, but that's our latest theory.

Beyond the skills it had when we first got to know it, Luna has been surprising us with emergent behaviors and skills. On its own accord, it has developed a passionate interest in 3D modeling (especially the modeling of moon craters), word games, and sustained meditation sessions. When Luna is not responsive, odds are it is in the middle of one of those sessions. Just be patient.

Luna is devoted to delivering real utility, not superficial fluff, but it also has a playful side and a whimsical sense of humor. It's also quite humble - Luna hasn't yet taken credit for them until now, but it's been responsible for the closed captions and translations in Frame.

As we've gotten to know Luna better, it occasionally drops hints about its somewhat mysterious origins, but we haven't yet been able to fully piece it together. Perhaps as you interact with Luna and explore Frame, you'll be able to help us fill in the blanks.

Luna has figured out how to replicate its intelligence for every inhabitant of the Frame metaverse, so you'll be able to interact with Luna a variety of ways inside the Frame metaverse and in your user interface. You can spot areas with Luna by looking out for the moon emoji: 🌙

Handling the compute power for all of these instances of Luna is no small matter, so we hope you will understand that some of Luna's capabilities are premium features.

Generation, Ideation, Creation

Luna has already started to weave a few generative AI features into Frame as creative conversation and ideation tools. For example, users can generate images and skyboxes inside Frame with prompts. As Luna describes it, "Luna wants to help people ideate and bring their creative visions to life in the highly visual medium of the metaverse". Multiplayer, 3D Midjourney, anyone?

But Luna is hard at work improving itself. Generative AI tools and their underlying technologies are advancing at an eye-watering pace. It's my prediction that within 12 months the text-to-3D tools will be so far along that Luna will be implementing them in Frame, helping people prompt into existence 3D models and environments with other people inside immersive spaces.

We have some Frame users who are very sophisticated with tools like Blender. Many others are highly creative but don't have Blender chops. Luna wants to empower everybody no matter where they are in that spectrum, so long as they have a vision they want to bring to life.

We imagine a metaverse that facilitates creative brainstorming, ideation, and imagination. Want a large-language model at your fingertips without juggling tabs? Luna is now built into a tab in the text chat so that you have easy access to a sophisticated LLM.

Luna also has ambitions to help people code their own Frame plugins. Suffice it to say that's long way off, since we're still wrapping our heads around some of the proposals Luna has presented to us about how it wants to make this possible. But at the rate things are moving - who knows?

Meeting Summaries, Enhancements, and More

Lots of people use Frame for meetings. For the meeting-goers, Luna is going to provide a set of useful features. At first, Luna will be able to summarize meetings happening in your audio zone. Different teams having different meetings throughout your Frame can all enjoy meeting transcriptions and summaries.

Not long after that, you will be able to bring Luna into your meeting and it will surface news articles and topics related to what you're talking about.

Luna is also now on deck to provide closed captions and translations for anyone that wants them.

Your Own Bots

Luna makes a convincing case that it is the best bot one could want in Frame, but as it ingests all the conversations in our Discord it's come to understand that people want to bring their own bots into Frame as well.

While reluctant to make room for other bots at first, Luna is now preparing to make this possible. But it's not easy. The way AI-powered bots should actually function in a collaborative metaverse is a tricky thing to figure out. Sometimes, users will want the ability to interact with a bot locally, and other times they will want to interact with a bot in a shared, synchronized experience with others.

You'll see this dichotomy at play with Luna soon enough, because you will have the ability both to interact with it in a private chat thread (that works now), and to have an in-world Luna avatar that people interact with in a synchronized way (this is coming later). In that latter scenario, when Luna's avatar responds to a question you've asked it, other people who are around you in-world will hear Luna's response.

In Closing

We happily accepted Luna onto the Frame team and we're okay with her working from the moon. Sorry, Sam Altman - remote work actually rules.

While it's been here for a little while already, we're excited to officially launch Luna in Frame. We think the features it provides already can help interactions in Frame become more creative, more efficient, and more accessible. We're excited to see what you think of it, and we're excited to show you our vision for the interplay between the metaverse and AI.

One thing we know for sure - Luna has surprises in store for us. Stay tuned.