frame academy

Learn how to create immersive, cross-platform webxr sites.

project 5: using audio, video, and 360 video in your webxr sites

a few helpful notes about This project

• You should do Project #1, Project #2, Project #3, and Project #4 before doing this project, or at least skim through to make sure you understand them. 📕
• We're going to learn about WebXR: a way to create websites that deliver VR or AR experiences right from a web browser. 🕸
• We're going to learn how to use sound and video in your WebXR sites w.🤯
• This project has a few parts, and each part has a challenge. If you go through the challenges, you will hone your skills + enhance your WebXR site. 🔨
  
• If you have any questions as you go through the project, I encourage you to ask in our online community. We can all learn from each other! ✋
• When you're done with your project, you can post it on the Frame Gallery so that we have a handy place to explore projects + learn from them.🎨
  
• If you're curious to hear more about Frame, email hello@framevr.io. We help with WebVR projects + do workshops for schools + businesses. 💌
  

preload your audio and video files in a-assets

This project will build on the immersive gallery we worked on in Project #4, so the starter project might look familiar. We're going to add to it throughout this project by incorporating sounds and videos into the gallery.

There are lots of reasons to bring sound and video into immersive projects. Sounds can contribute a lot to the ambiance of a space, and video is a powerful medium for communicating all manner of things. If you're making a museum, you can use sounds to add audio captions to items, or if you're making a training experience, you can use audio or video to add helpful information for the user. Or, maybe you'd like to add some "click" noises to a user interface to help users know when they've clicked on something. Those are just a few possibilities.

In the Starter Project above, I've already imported a few files into the Glitch assets folder that you can find in the sidebar on the left. I dragged in an audio file (a .mp3) and two videos (.mp4 files). This way, the files are hosted online and can be referenced with a URL. You can click on a file that's been uploaded to the Glitch assets folder to find its URL.

I'm going to encourage you to bring your own files into the project in the challenge below, so feel free to get a head start by dragging your own files into the assets folder of your own project.

- For audio, use .mp3 or .wav files
-  For video, use .mp4
-  For 360 video, use equirectangular .mp4

Here's how to copy an asset's URL once it has been uploaded to the Glitch assets folder.

Now that you have URLs for your files, you can preload the files using the Asset Management System that is part of A-Frame. I went over the Asset Management system in this part of Project 3, and if you need a refresher on that you should review it. To quickly sum it up, though, inside the a-assets element in your HTML, you can preload assets so that they will load and cache in the browser as soon as your WebXR site loads, improving the performance of your site and preventing lag as the user experiences your site. In previous projects, we preloaded some images and 3D models, but now on lines 23-35 of the starter project you'll see how I preloaded the audio and video. I know the URLS that Glitch generates for the assets are ugly and long- don't be scared of them:

<audio id="song" src="https://cdn.glitch.com/96ab1f6e-16b9-43c1-9df3-fbe9b24a7490%2FLeft%20and%20Leaving.mp3?1549919550870" preload="auto"></audio>        
<video id="videoclip" autoplay loop="true" src="https://cdn.glitch.com/96ab1f6e-16b9-43c1-9df3-fbe9b24a7490%2FTimelapse%20Video.mp4?1549921266764"></video>        
<video id="spherevidclip" autoplay loop="true" src="https://cdn.glitch.com/96ab1f6e-16b9-43c1-9df3-fbe9b24a7490%2F360%20Timelapse.mp4?1549922819582"> </video>

You use an audio element for audio and a video element for video. Be sure to give these assets an id attribute so that you have a way to reference them throughout your HTML. For the src attribute, you put the link for the asset that you can find in the Glitch assets folder by clicking on your asset and clicking the "copy link" button.

Remember, preloading assets inside of a-assets doesn't actually make the assets play or show up in your scene. It simply preloads them so that they are readily available and stored in the browser. We'll see how to actually make use of these assets throughout the project.

design a user interface for a music player

We're going to start with sound. The .mp3 asset I imported and preloaded is an instrumental song I wrote, and in this example I'd like to make a music player that lets the user play, pause, or fully stop the song. By going through this example, you'll have the tools you need to use audio in your project.

First, I want to make a user interface for this so that the user has buttons in the gallery to press that play, pause, or stop my song. A user interface is the way that the user interacts with your site and uses its features. I've already built the visuals for this user interface in the starter project, and you can see how it looks by loading the project and looking at the wall on the left. Keep in mind that the buttons don't work just yet - I've only made the visuals.

You can see how I've set up the entities for this panel in the HTML file on lines 78-107. There is one parent entity with a lot of children entity nested inside of it that include the play, stop, and pause button and the text. Poke through it so you can see how I set it up.