Apple and the Spatial Web: A Winding Path Towards a Bright Future

Two and a half years ago, I wrote a blog post titled "The Immersive Web and WebXR Matter More Than Ever". In that post, I wrote:

Apple, for one, seems invested in keeping iOS Safari just bad enough so that people are forced into using mobile apps to access any powerful experience. Apple likes this because it means more $$ from app-derived revenue. So, it's no real surprise that Safari doesn't yet support the WebXR API, although there are rumors that this is in the works. As Apple readies their own immersive hardware, it remains to be seen whether they want Safari to be an actual gateway for 3D experiences, or if they will push everything into native applications.‍

Well, now we've seen. Apple announced their Vision Pro headset, Safari is an application on it, and the WebXR API will be supported on it out of the gate. Sweet.

There are still open questions about Apple's approach to the WebXR standard and the web at large that are murky to me. At a high level though, Apple letting Safari participate more meaningfully in the immersive web is a big deal ... for the web generally, "the metaverse", and for Frame.

Let's dive in to a bit of history that led up to this being such a welcome development.  At a high level, I have my complaints with Safari, but I'm cautiously optimistic about it and I was thrilled by what happened at WWDC this year.

WebXR Basics (skip if you know what WebXR is)

First, for those unfamiliar, what is WebXR? In a nutshell, it's browser API that lets web sites interface with immersive hardware. An example of this is the browser on the Meta Quest headsets, which lets websites render in immersive mode and receive input from the headset and controllers.

There are many reasons why this is exciting. For starters, WebXR means that web developers can craft immersive experiences without necessarily relying on proprietary game engines. Because the browser is the target, this also means that the same code base can target many platforms at once, with variations for different input mechanisms.

When Meta carefully curated which native apps showed up in their Meta App Store, Frame was able to work on the Meta Quest and Quest 2 the very days they came out, because it ran directly from the browser - a browser that Meta, to its credit, works hard to make compatible with the WebXR spec.

Apple and the 3D Web (skip if you only want to read good things about Apple)

WebXR aside, it is simply not easy to make sure Frame runs successfully from Safari on either desktop or mobile. We aren't alone in thinking this. This is unfortunate, because there are things I like about Safari such as the way they prioritize user privacy.

That said, in the area of "all things 3D on the web", it's pretty reliably bad 😬: iOS updates regularly break 3D experiences, Apple is notoriously late in their support of web standards that drive powerful 3D experiences, and their historical nonchalance towards things like PWAs (progressive web applications that look and feel like native apps), suggest that philosophically Apple is uninterested in the idea of a website that can reasonably compete with native applications.

They have good reason for this attitude, given how much revenue they derive from the App Store.

Another fundamental building block that enables high-performance, sophisticated 3D experiences on the web is WebGL. Apple was five years late to support WebGL2 compared to Chrome, and their support of it is still woefully buggy and unreliable.

Meanwhile, the next generation of WebGL, WebGPU, has already shipped in Chrome. There is little word from Apple on their timeline for WebGPU. Early support for WebGU was tucked behind unreliable, experimental flags in Safari versions from four years ago, but they no longer are.

(WebRTC is a whole other story, which I'll probably save for another blog post, but the tl;dr is that WebRTC is used for audio/video/data communication through the browser, and not only was Safari really late to have good WebRTC support, but even after they supported it they limited key capabilities of WebRTC in non-Safari browser applications on iPhone.)

With WebXR, Apple has confusingly opened up some aspects of the API behind Safari experimental feature flags. Sadly, the flags don't seem to do anything when turned on. On the other hand, Android has supported WebXR for years, letting developers build immersive augmented reality experiences that run from a smartphone, using the phone's camera to provide an overlay onto the real world.

Apple doesn't allow other browser engines to run on their hardware, so it's Safari or bust. Chrome on iOS is just a skin on Safari. If that sounds strange to you, you aren't alone. It's also strange to regulators across the pond, where lawsuits are winding their way through the courts that see this as anti-competitive behavior on Apple's part. In preparation for a world where Apple lets other browser rendering engines on iOS, Google and Mozilla are already working on iOS versions of their browsers using their own rendering engines instead of Safari's Webkit.

It's worth pointing out, though, that Apple appears to be preparing Safari to be more competitive, even inching towards better support for PWAs - the holy grail for web developers who want to make great software that can be accessed outside of any particular app store. For example, Apple recently launched Web Push support for iOS Safari, letting PWAs give notifications on mobile, and more recently at WWDC they announced much better PWA support for desktop Safari, letting users "save to dock" websites so that they can look and feel more like a native application.

These are tangible steps in a great direction for Safari, but a lot more has to be done for Apple to solidify broader developer trust and excitement about it.  It's not like developers WANT Chrome to be the only good browser. It's just that it's not easy to develop around Safari's quirks, hit-or-miss browser standards support, and generally poor communication around its roadmap and priorities.

For 3D developers, it's been a particularly painful, confusing journey. Many of our user-reported problems at Frame are from people who have performance issues on the Safari browser, particularly from iPhones. It's stunning to me that five year old Android phones can handle many 3D websites better than the latest iPhones.

With WebXR, it's been confusing because over the past few years, Apple has hired really top-notch WebXR talent from all over the world.

For a look at how it can be argued that Safari is holding back the web generally, take a look at "Progress Delayed is Progress Denied" post from Alex Russell.

For a more in depth exploration of how developing 3D web sites that reliably work on Safari is hellishly difficult, check this blog post from Construct.

Apple Vision Pro and Spatial Safari (don't skip... the metaverse is back, baby!)

Don't let my web-centric rambles above give you the wrong impression - most VR developers don't care much about the browser as a vehicle for 3D experiences. Most build native applications with Unity or Unreal, which I understand. More on this later.

So, skip to WWDC 2023. I've tuned in and I'm waiting for any mention of Safari or WebXR in the context of Apple's big reveal: the Vision Pro. I'm also curious to know more about how Apple imagines the role of the web in its vision for "spatial computing".

I was a little disappointed in the keynote.  Safari was in the Vision Pro marketing videos, but it was someone pulling up Safari in their living room to view some regular web content (see image above). To me, this alone IS sort of exciting stuff because I'm interested in the prospect of getting rid of every monitor in my apartment and replacing them with glasses that can overlay everything I need on top of my view of the real world. But from the WebXR perspective, this was a disappointment.  Safari didn't power a 3D immersive experience, and in the discussion about Safari in iOS 17 there was no mention of WebXR. Heck, it wasn't even tucked into this jam-packed slide of new things that would be in Safari 17.

I took to Twitter to lament the let-down even though I've experienced some version of it dozens of time before when watching Apple events or looking at Safari release notes.

But hope springs eternal for the immersive web. The next day at WWDC, there was a dedicated session on Safari called "Meet Safari for Spatial Computing." You should watch it here.

In this session, Apple revealed that Safari will indeed support the WebXR standard when the Vision Pro launches, and developers can create full 3D experience that users can interact with using their hands via the WebXR hand-tracking API. This is a big deal.

They even had a slide that mentioned Babylon.js, the particular open-source web development framework from Microsoft that we use to build Frame!‍

Because of this, Frame will be accessible on the Vision Pro at launch.

Caveat: Apple's WebXR support on Vision Pro will only be for full VR sessions and not AR sessions, pending discussions and work related to the WebXR API specification (Apple is actively involved in these standards discussions). There was also no mention of WebXR on the iPhone, even though many developers have been waiting to build AR experiences that work from Safari on iPhones.

Impact

To take a step back, the Vision Pro is a veritable wonder of display technology and computing power. Apple will most likely do to this general category what they've done to many others.

So, to be able to develop immersive experience for it with WebXR means that the world of web developers will be able to target this extraordinary piece of hardware and make the next generation of spatial websites. Apple may be changing the way they are thinking about Safari, and I'm cautiously hopeful for more in this direction, a direction that empowers all web developers to build beautiful, performant, and functional sites.

I also think Apple will support WebGPU sooner than expected, which will help Safari more efficiently leverage the chips inside Apple hardware.

Apple is prudent to unlock a spatial Safari. In some ways, this hearkens back to the original iPhone when Safari was the early vehicle for most content on the first iPhone and there was no App Store.

I'm excited for when millions of websites will have a spatial dimension that kicks into gear when the site is viewed from a WebXR-enabled device.

I've long thought that the web browser sort of IS the metaverse, or at least a huge part of it. It's the connecting platform for disparate worlds that users already know how to use. The site of all sites.

Closing

Regarding Unity and Unreal, two of the big game engines for making 3D content, it was notable to see Apple announce a Unity partnership without a single mention of the Unreal Engine from Epic Games. This is despite the fact that the Unreal Engine is considered by many to have leapt ahead of Unity in recent years. There's a lot of speculation about Apple's decisions around this, but it seems likely that Epic's ongoing legal wrangles with Apple aren't doing Epic any favors in this area.

I mention this because it highlights one of the benefits of working with open web standards - Frame is not at the whim of the fortunes and vicissitudes of particular game engines or proprietary technology.

We are indeed at the whim of those that create mainstream web browsers. In this case, when the collective capabilities of the major browsers take a big step forward, it raises all ships on the web.

We're excited to bring Frame to the Vision Pro headset through Safari.

We're also excited about the Meta Quest 3, and I'll be posting more about that in the weeks ahead.

Grab your own multi-user, spatial website: https://framevr.io