Frame December 2024 Update: Splats, Performance, and More

Frame is the easiest way to come together with others in an immersive space to collaborate and create, right on the web. We have a bunch of sweet new features to show you. Some of them set the stage for our broader vision to bring AI to immersive meetings, but we also have new assets you can add to your Frames, performance optimizations that help your Frame run smoother, and a neat new way you can give tours of your Frame.

Splats Are Here!

Gaussian Splatting is a powerful new way to create 3D scenes. By taking just a few pictures with your smartphone, you can generate incredibly lifelike 3D models with detailed lighting and reflections.

Instead of using lots of tiny triangles like traditional modeling methods, Gaussian splats use soft, colorful blobs to fill space. These blobs work together to form shapes or scenes.

Note that splats can still have the same problems of traditional 3D models - if they are too complex they can cause performance problems in your Frame.

We expect lots of improvements to keep happening in this area of 3D scene rendering, not just with Gaussian Splatting but with other rendering techniques too.

I've been using an app called Polycam to capture some splats of my own, and I found this one in their library that I dropped into Frame:

You can't yet incorporate a splat into your navigation mesh, so the movement around the splat is really free-form. For example, you will walk through walls etc.

Generally speaking, I think Splats are best for smaller, individual models though - not entire environments. They are especially compelling for things that have delicate, vibrant textures (plants, etc).

Soon, we'll also support Niantic's new Splat format: spz. They call it "the .jpg for 3D Gaussian Splats".

Follow the Leader

You can now have other users follow you around your Frame automatically. This can come in handy if you want to give a walkthrough of your Frame without making your guests manually move around the space. I think this will be particularly useful for those who use Frame to show off places and properties, because it removes all the friction that can come with getting people to the same place. Herding cats can be hard in the metaverse! (Note: you can of course always use the "gather to me" button to bring everyone around you, but that's not as interactive as having everyone move around the space together).

Performance Improvements

We know that Frame users often want to bring loads of assets into their Frames - images, 3D models, streaming screens, and a lot more. While we generally encourage this, we also do our best to make clear that adding lots of assets to a Frame can impact its performance - especially on lower-end hardware, mobile devices, or VR. All that said, we've introduced some strategies this past year that can help you find this balance between the asset-rich, vibrant Frame you want and the smooth performance you need.

For example, on image assets we have a "hide at a distance" switch you can find if you select the image asset in Edit Mode. This is a simple but powerful strategy that ensures that far-away images won't get rendered, but then will show up as you move closer to them. We also have a "make private to zone" switch which makes it so that assets inside of a voice zone only show up for users who are also inside that zone.

In this release, we're happy to bring the "hide at a distance" switch to many more kinds of assets, like text areas, polls, videos, stock tickers, pdfs, whiteboards, and more.

In addition to that, we've also done some key work under the hood to improve the rendering performance of many of our assets, including text labels, streaming screens, and others.

Our message is the same as it's always been - while we will do our best to have Frame run as smooth as we can, it's up to you to find the right balance between performance and asset-richness. Where you land on that spectrum depends on how you take advantage of these optimization strategies and the requirements of you and your users.

AI Video Generation

We put this out in a small release a few weeks ago, so I thought I would give it a bit of love in this blog post. You can now ask an AI agent to create videos for you and it will generate 5 second clips, either using a text prompt or an image reference. Here's a look:

AI Agents

We have had "NPCs" (non-player characters) in Frame for a while as an experimental way to bring some AI features into Frame. We've been evolving these NPCs quite a bit under the hood, and while we don't have much to speak of just yet, we have rebranded them to "Agents", and you will see a new "AI Dashboard" linked right from the sidebar and other parts of the UI. In the AI Dashboard you can modify your agents, view your meeting transcripts, and add things to a knowledge base that all your agents are trained on.

While we encourage you to experiment with the knowledge base, please note that in our next release we anticipate that you will need to re-upload any files you want to have in your AI knowledge base. So if you add things to it now, please consider this only for short-term experimentation purposes. You can also upload files to specific Agents to give them their own special knowledge base, but the same note applies for those files - you will need to reupload them after our next release. We appreciate your patience as we make the moves we have to in order to establish a rock-solid agent architecture in Frame.

We are only getting started with the AI features in Frame. Broadly speaking, I think AI has yet to come to meeting software. In the same that we are seeing developer tools get rewritten with AI in mind to boost developer performance, meeting tools will also be refashioned in powerful ways to incorporate AI and make meetings more creative and productive.

Beyond that, the AI agents in Frame will soon be able to do work in between meetings, helping you prepare for the next one and contributing as valuable players on your team.

In Closing

We hope you take advantage of these great new features. Stay tuned for more, and let us know what you'd like to see in Frame next!