frame academy

Learn how to create immersive, cross-platform webxr sites.

project 5: using audio, video, and 360 video in your webxr sites

a few helpful notes about This project

Project goals

I hope that by going through this small project, you will learn:

part 1: importing your assets

setup your project

Each part of the project has a video at the top where I guide you through the project. The video is the best place to start, but I also break down the content below so that you can follow along step by step.

To code our projects, we're going to use something called Glitch, a sweet tool that gives you a way to write your code from a web browser. Glitch also hosts your website for you, letting you see the results of your code immediately!

I made a starter project that you should use as the base for this project. You can make your own copy of it with this button:

copy the starter project on Glitch

You can also "Remix" any of the code samples you see throughout the project by using the "Remix to Edit" button you'll see next to them. This lets you make a copy of my code that you can edit, build upon, and make your own. You can also always copy and paste my code into your own project..

To save your Glitch project so you can keep working on it whenever you want, you have to sign in to Glitch with a free account. The sign in button is on the top right of Glitch.

Reminder: Have a question or idea? Check out the Frame Academy Community on Discord.

preload your audio and video files in a-assets

This project will build on the immersive gallery we worked on in Project #4, so the starter project might look familiar. We're going to add to it throughout this project by incorporating sounds and videos into the gallery.

There are lots of reasons to bring sound and video into immersive projects. Sounds can contribute a lot to the ambiance of a space, and video is a powerful medium for communicating all manner of things. If you're making a museum, you can use sounds to add audio captions to items, or if you're making a training experience, you can use audio or video to add helpful information for the user. Or, maybe you'd like to add some "click" noises to a user interface to help users know when they've clicked on something. Those are just a few possibilities.

In the Starter Project above, I've already imported a few files into the Glitch assets folder that you can find in the sidebar on the left. I dragged in an audio file (a .mp3) and two videos (.mp4 files). This way, the files are hosted online and can be referenced with a URL. You can click on a file that's been uploaded to the Glitch assets folder to find its URL.

I'm going to encourage you to bring your own files into the project in the challenge below, so feel free to get a head start by dragging your own files into the assets folder of your own project.

- For audio, use .mp3 or .wav files
-  For video, use .mp4
-  For 360 video, use equirectangular .mp4

Here's how to copy an asset's URL once it has been uploaded to the Glitch assets folder.

Now that you have URLs for your files, you can preload the files using the Asset Management System that is part of A-Frame. I went over the Asset Management system in this part of Project 3, and if you need a refresher on that you should review it. To quickly sum it up, though, inside the a-assets element in your HTML, you can preload assets so that they will load and cache in the browser as soon as your WebXR site loads, improving the performance of your site and preventing lag as the user experiences your site. In previous projects, we preloaded some images and 3D models, but now on lines 23-35 of the starter project you'll see how I preloaded the audio and video. I know the URLS that Glitch generates for the assets are ugly and long- don't be scared of them:

<audio id="song" src="" preload="auto"></audio>        
<video id="videoclip" autoplay loop="true" src=""></video>        
<video id="spherevidclip" autoplay loop="true" src=""> </video>

You use an audio element for audio and a video element for video. Be sure to give these assets an id attribute so that you have a way to reference them throughout your HTML. For the src attribute, you put the link for the asset that you can find in the Glitch assets folder by clicking on your asset and clicking the "copy link" button.

Remember, preloading assets inside of a-assets doesn't actually make the assets play or show up in your scene. It simply preloads them so that they are readily available and stored in the browser. We'll see how to actually make use of these assets throughout the project.

design a user interface for a music player

We're going to start with sound. The .mp3 asset I imported and preloaded is an instrumental song I wrote, and in this example I'd like to make a music player that lets the user play, pause, or fully stop the song. By going through this example, you'll have the tools you need to use audio in your project.

First, I want to make a user interface for this so that the user has buttons in the gallery to press that play, pause, or stop my song. A user interface is the way that the user interacts with your site and uses its features. I've already built the visuals for this user interface in the starter project, and you can see how it looks by loading the project and looking at the wall on the left. Keep in mind that the buttons don't work just yet - I've only made the visuals.

You can see how I've set up the entities for this panel in the HTML file on lines 78-107. There is one parent entity with a lot of children entity nested inside of it that include the play, stop, and pause button and the text. Poke through it so you can see how I set it up.

Tip: User interface design is an art of its own, and I'll admit I'm not great at it! Look around at some sites or apps that you think are both beautiful and easy to navigate to get inspiration for making your own user interfaces. There are people who specialize in user interface design, and for good reason. It doesn't matter if your site has loads of sweet features if it's difficult to use them!

🔥 challenge #1 🔥

Each part of this project comes with a challenge. As you go through the challenges, you'll be building a WebXR site that will work on desktop, mobile, or VR!

For this challenge, start to make the starter project your own. Some ideas? Add your own images to the walls, swap mine out, bring a new photosphere in for the sky, bring in other 3D models, and arrange things however you want them. For a refresher on how to bring in those assets, check out Project 1. Start making a cool gallery about something you love, or an e-portfolio that shows off some of your work, or just an exploratory playground.

Move on to part 2 >
playing, pausing, and stopping audio