Frame at Microsoft Build: Speaking the World Into Existence

We believe that creating virtual worlds in the metaverse should be accessible to anyone who has a vision they want to bring to life. Most virtual worlds are created with sophisticated game engines like Unity3D or the Unreal Engine, tools that can be hard to wield.

Speaking the World Into Existence

We were proud to collaborate with Microsoft on a prototype for Microsoft Build that showed off AI-powered creation using natural language input.

In our prototype, you speak or type to an AI-powered bot inside Frame, and the bot manipulates the 3D scene, doing the best job it can with your commands to bring your vision to life.

Babylon Codex

The AI system that powered this experience was a custom version of OpenAI Codex called Babylon Codex, specially tuned to take natural language commands and output code in Babylon.js, the open source 3D development framework from Microsoft that we use to build Frame.

We had it adjust the gravity of the physics engine to match the gravity of the moon, which it was able to determine without any training from us.

This was still just scratching the surface of what can be done with these tools, and we'll continue to experiment with Babylon Codex.