Translation and Closed Captions in the Metaverse

At Frame, we're building the easiest way to create your own corner of the metaverse for collaboration, education, events, and more. You can use Frame to create your own 3D destinations without writing a line of code, then invite people to them just by sharing a link. Each Frame is like a website, but it's immersive, super-powered, and collaborative.

We've been working on making Frame more accessible - this means a lot more than just being available from the web. It also means making the experience useful and functional to as many users as possible. Closed captions and translation in Frame are two recent features that help us on this mission. This post will talk about how they work and why they matter.

They also work together nicely, letting you translate other people's speech in real time and see the output in your closed captions!

We have a long way to go and a lot more work to do in this area, but the metaverse should be for everyone.

Here's a look at it in action (along with a few other updates):

Closed Captions

Perhaps most importantly, closed captions provide a way for users who have hearing disabilities to be able to get a sense of what people are saying. Closed Captions are also helpful for those who process language better when reading it as opposed to listening to it. Closed Captions in Frame aren't guaranteed to be perfect, but they are solid across a wide range of languages. Here's how they work in Frame.

If you're logged in to Frame, you have a CC button in your toolbar at the bottom of the screen that you can use to activate Closed Captions. When you click this, if there are users in the Frame who haven't yet given permissions for the browser to access their speech to try to transcribe it, those users will see a pop up that asks for permission. Once users grant this permission, then any user that has requested Closed Captions will get those users' speech to appear in a closed captions display at the bottom left of the screen.

Note: The reason why we ask for this permission is because our closed captions feature is powered by the Web Speech API. This API relies on external servers to power the speech-to-text functionality. From Chrome or Edge, this is powered by Google. Safari uses Apple. While your audio isn't stored on these servers, or used for any other purpose besides the transcription, we nevertheless wanted to explicitly ask for permissions from users before letting their speech be processed.

You will only get the closed captions for users who are in your voice zone, so be sure to strategically set up voice zones if your Frame is large and you don't want users getting closed captions for other users who are really far away. Frame lets you set up voice zones wherever you'd like.

You will also see the closed captions in the text chat, just in case you want to download a transcript of the meeting later that includes all of the closed captions.

For users that aren't speaking English, they may be able to improve the accuracy of their closed captions if they explicitly set their Voice Language in the Profile section of the sidebar. This is where you can also set your own Translation Language...more on that below!

Right now, closed captions just work in desktop mode, but we'll be bringing them to VR and mobile as well. It works best in Chrome or Safari.

Translation

Lots of meetings in Frame include people who speak and write different languages. You can try to agree on a lingua franca and get by, but you can also just let people talk in their native language and turn on closed captions and pick a translation language. This will give you translated closed captions. People can set their own translation language to get the captions in the language of their choosing.

Turn on closed captions with the CC button in the toolbar at the bottom of the screen as described above, and pick a translation language in your Profile in the sidebar (see screenshot above).

Picking a translation language will also translate the text chat for you when people write text messages in the chat.

Just like with closed captions, translation isn't guaranteed to be 100% accurate. We've seen it be pretty darn good, though. You can also see the original, untranslated words by hovering over the message, either in the text chat or the closed caption!

Here is me getting some Finnish translation in the closed caption when I speak English.

These two features on their own are both powerful, but when combined let you have other people's spoken word translated for you in real time.

We have a lot more coming to make Frame more accessible to more people around the world.