Transcript

Jen

[00:00:00] This is The Web Ahead, a weekly conversation about changing technologies and the future of the web. I'm your host Jen Simmons and this is episode 96.

I want to first say thank you so much to Pantheon for running the platform for our new website. You can check them out at getpantheon.com. You can get out the new website at thewebahead.net. Bandwidth for this episode has been provided by CacheFly, the fastest and most reliable CDN in the business, at cachefly.com.

Today 'were going to talk about funky new technology that may or may not be the newest, hottest, coolest thing on the web in, like, five years. [Laughs] Sometimes we do these shows that are very speculative, finding out about the real cutting-edge and where people are inventing new things and playing around with new things. Today I have two guests on the show from the Mozilla Research Labs, messing around with virtual reality technology and WebVR, figuring out what the web might be doing with VR.

One guest is Josh Carpenter. Hi, Josh.

Josh

Hi Jen.

Jen

And Vlad Vukicevic. Welcome to both of you.

Vlad

Thanks very much for having us.

Jen

[00:01:30] So tell us about what you're working on. The project and the team. You're over at Mozilla. What are you doing? You're in some secret lab behind a big, giant set of doors that you have to have special eyes? You have to have the right eyes to go through the doors?

Vlad

[00:01:44] No quite so secret. We're Mozilla, so most of what we do, we try to do as much in the open as possible, because we really value the collaboration aspect of it.

About six months ago, I was looking for a new project and Josh was as well. We both independently set our eyes towards VR, virtual reality. We wanted to figure out, how do you do VR on the web? Does it have a place on the web? If so, how does browsing look like on the web? How do people create content? How to do they put content on there? Does the current web transform into VR? Do we have to abandon it and start over? What are the technological pieces that are missing? What has to evolve from the web platform? What is needed in CSS and new APIs and all of that?

We both started talking and decided this was something we wanted to focus on. We got the approval from our CTO and we both jumped in with both feet. Since then, it's taken off quite a bit. We have a lot of interest, both from other browser vendors and a lot of other web developers, helping us figure out what this VR thing is.

Jen

[00:02:51] Is there a specification yet over at the W3C? A draft? Or are you working towards such a thing?

Vlad

[00:02:57] We're sort of working towards such a thing. We want to make sure that we don't specify anything prematurely because this is still very much a research project. We're figuring out what pieces the web actually needs here. Myself and a collaborator at Google, Brandon Jones, who is doing a lot of the Google-side implementation of all of this, we just recently — literally in the last 24 hours — started putting together a draft specification of the bits and pieces we're adding now. But it's not part of any standards body at the moment. It's really just more of a collection of notes of what we're experimenting with currently.

Jen

[00:03:31] Yeah. So talk about virtual reality. I think when most people think of VR, they think of headsets that you wear playing games, certain kinds of fully immersive video games. What are people using VR for? Is that what you're doing? You're figuring out ways to put video games on the web in a new way?

Josh

[00:03:51] I was just listening to someone else's podcast last week and I think they said that VR is just games. I think we look at it as, it is very much a general purpose medium. It's a new tablet. It's a new television. It can be anything you want it to be. It's display and input technology. It kind of makes sense that games are jumping into it first. If you think about it, "Who has the skills to actually create 3D environments?" It's game designers — and visual effects designers, for that matter. And also, "Who has the inclination to buy a technology like this to be an early adopter?"

It really is going to be gamers. But we're already seeing a bunch of applications outside of the games realm. For example, I was reading about neuroscientists who are trying to help soldiers who might be afflicted with PTSD. Actually try to get past that using exposure therapy. By taking them into the environments in which they encounter trauma and helping them work through that trauma.

Filmmaking is another big one. I think Oculus just announced a filmmaking studio, internally, to explore storytelling in virtual reality.

Really, the way that we look at it is, the kind of magic of virtual reality as a platform is going to be presence. This is the unique ability of virtual reality to trick your brain into thinking that you are actually in a place. If you're a thousand feet up in virtual reality, you actually, your body actually believes it. You get sweaty palms, you get shaking knees. There's nothing else quite like that. That fundamental magic, that applies to anything you'd want to apply it to. From education to titillation to gaming. Really anything. We see it very much as a general purpose technology.

Jen

[00:05:32] Do people do VR things without a headset, some sort of headset? Or is the headset always part of it?

Josh

[00:05:43] Headsets are definitely always part of it. If you want to think about it, there's virtual reality and there's augmented reality. It kind of exists on a spectrum with mixed reality in the middle.

In virtual reality, everything you can see is defined by the screen. You can only see the screen. There's no actual reality being shown to you.

Whereas, in augmented reality — like Google Glass — it's exactly the opposite. You're seeing mostly reality with a thin overlay of digital information.

Mixed reality is kind of in the middle. It's a bit like Microsoft HoloLens. With mixed reality, the headset will actually have the capability of including something in the real world and replacing it with something digital and 3D. For example, you're looking at your car, but your car's been removed, occluded by the headset and replaced with the car you'd like to have. Or with a rendering of the engine as it should be working, not broken.

VR, AR, and MR all require headsets. Right now, these headsets look like having a lunchbox on your face. [Jen laughs] I think a lot of people have trouble getting past that. If there's one thing you can bet on with technology, it's miniaturization, increasing performance and decreasing cost. We've seen the projections going out three years or so, in terms of what the technology is going to look like, from a form factor standpoint. It gets into Ray-Ban territory really, really, really fast. Which makes it a heck of a lot more palatable to the average consumer.

Vlad

[00:07:14] One of the main goals with VR is obviously to transport you elsewhere, like Josh was mentioning, presence. The headset, the ability to take over what a person sees, to take over their visual experience, is definitely a critical portion of it.

But the other senses are also important. What the user hears is — in some cases — even more important. If you were trying to experience walking through a forest scene, you really want to hear all the forest noises around you. If there is a deer to the right of you — which, thanks to VR, you can actually turn your head to the right and look at — you want to actually hear it walking by. So you have a cue. Audio becomes very important. Even the other senses. At some point down the line, you might get smell. When you're in the forest, you should be able to smells the firs. You should be able to smell the grass. It's really about taking over as much of the sensory experience as possible, and directing it in a particular way, based on what experience you're trying to recreate.

Jen

[00:08:18] As you figure out how this could happen on the web, is it really about figuring out some kind of open standard to make VR systems interoperable and just transmit all of that data across the internet? Or is there another layer? That VR on the web would be different than anywhere else, because it's on the web?

Josh

[00:08:40] I think it tends to be the latter. Figuring out the standards that will enable this is obviously an important part of it. Let's say you get an Oculus headset in a year. You fire up the home screen. Let's say there's a browser on that home screen. Right next to a $400 million title from EA and a bunch of movies and games. Why would you click on the web icon, especially if you've got a retina laptop and a smartphone and a tablet, which all render websites beautifully, with high resolution? What's going to be compelling about the web with virtual reality? I think that's what we have to figure out as a community.

A good place to start — and we tried to do this in Firefox OS as well — is looking at the strengths of the web and what makes the web unique. Such as interoperability. That huge library of existing content, such as the ability to go anywhere from anywhere. No matter where I am on the web, I can type in a URL can I get anywhere else. That's extremely powerful. That's a huge differentiator from what we see right now in native VR, where I have to install these applications and who knows what's inside the application. I can only get the application through a store. The web can be that interoperable, enabling, go-anywhere-from-anywhere layer.

I think we'll take those basic qualities. We'll take some of the existing content and sites of the web — like Wikipedia, Reddit, Flickr. Major properties that have a lot of content and mindshare. We'll try to figure out, "What does Wikipedia look like in virtual reality? Why do I go to Wikipedia?" What is the job to be done of the web? What is the job to be done of Wikipedia? How can we marry that job to the unique power of presence? That will be a really, really exciting process for the web community to go through. Figuring that out.

Jen

[00:10:35] Yeah, it's a weird question. It feels very 1980s to me. This idea that somehow you would climb into the machine. [Josh laughs] There are all of these movies from the '80s, right? With all of this tech-future-music and tech-future-fonts.

If you think about it, the late '70s and early '80s were this moment in time. Computers had been around for awhile, but they were very specialized. It was like, "How many people hang out in an operating room at a hospital?" Very few people. "How many people use a computer?" Like, about that many people. [Laughs] It went to, "Ok, let's have a computer in our house." People were like, "What are we going to do with these things called computers? Why do we want one of those in our house?" There were all of these movies where it was like, "It will be the future and you'll be inside the machine! It will be really awesome!" It feels like, Wikipedia and virtual reality? So I'm going to climb inside of Wikipedia and walk around and read the content on the walls of the world as I'm experiencing this virtual world.

There is something about it that seems very, like, "Yeah, that's never going to happen. I don't care about that."

Vlad

[00:11:48] I think that's actually one of the things we're really trying to figure out. Because we don't really want that kind of experience. Sure, if someone wants to create the Wikipedia where the text is written on the walls, that might be interested in some areas. But, as you say, it's not a compelling vision.

For Wikipedia, instead, imagine if you could go to a Wikipedia article that's presented in some way — whether that's in VR or not — but then when you're talking about a place or an event, you're able to experience that event. You're able to jump in and have that presence, without leaving your web browser. Without having to download an additional video or special player or any of that.

One of the things that attracted both Josh and me to this was... science fiction has, for a long time, talked about this notion of virtual worlds, interconnected worlds, being able to — as you say — jump into the machine. We already have a lot of that with the web. We already have this massive, interconnected network that's distributed, not controlled by anybody. Very low barrier to putting your own content and creativity out there. We wanted to make sure that we were able to take advantage of that. How do we actually pull that into this new virtual reality world instead of trying to recreate it?

I think the Wikipedia of the future or the Wikipedia of VR will really involve augmenting the content with that feeling of presence, rather than trying to have you jump into Wikipedia or walk around Wikipedia.

Jen

[00:13:20] Yeah. It'd be amazing. "Oh, this little town in Italy, I can go there."

Josh

[00:13:25] That is a magic of the technology. Imagine you had never heard of virtual reality. You hadn't heard of the sci-fi. If I told you that in a year, for $300 you could buy a lightweight headset that would transport you anywhere in the world, anytime in the world, and you could be anybody or anything, and that you would believe it. That would be as bad as it was ever going to get. Each year it would just get better and better and more realistic and more realistic. That's what's coming. That's the crazy part. Sony, Microsoft, Facebook, Google. Each of them have announced display-computing technology. It's such a surreal thing.

A lot of us in the VR community talk about this. The only metaphor I can come up with is, it's like knowing a big astroid is about to hit the planet and trying to walk around and tell people about that and not sound like a crazy person. [Laughs] It really is a technology you need to try, to get.

We had the opportunity at Oculus Connect — which was a convention put on by Oculus in September — to try their next generation Crescent Bay prototype. This is a prototype of what will come, probably towards the end of this year, form Oculus. I've got to tell you. Their first demo, they drop you, as kind of a Batman-character, sitting on a ledge of some steampunk city a thousand feet in the air. You believe it. And in the next scene, you're being chased down a hallway by a dinosaur. And you believe it. In the last scene, you're in a slow motion battle scene with cars flipping all over you. And you believe that, too. It's pretty remarkable.

I think one of the challenges for VR will be trying to get as many people as possible to try it. Once you try it, you believe it, and I think we'll go from there.

Jen

[00:15:10] How great is the quality of the visual that you see on the new headsets that are coming out? You say you believe it. I guess I still have in my mind, early VR stuff that I looked at. Not even headsets. It wasn't really VR. It was, like, Director Shockwave videos, or QuickTime 3D, and you sort of spun around. [Josh laughs] This was years ago. The quality was just so bad, and it left me with this impression. Or I go to a 3D movie and it's dim and hard to see the screen. It's like a lower quality version of the movie and sort of 3D. It leaves me very skeptical. What is it actually like?

Josh

[00:15:56] QuickTime VR was awesome, by the way. In 2003 I was doing projects with QuickTime VR. It was so sweet.

Right now, there's kind of two tracks. One is film. There's a bunch of people working on 360° stereoscopic camera. They can essentially record a scene in extremely high resolution — usually 60 FPS or higher in 3D. Then you watch that footage back in the headset. In fact, at Sundance right now, there's a Sundance-cordoned off part of the festival for virtual reality short films. There's a bunch of really talented filmmakers experimenting with this technology.

The beautiful part of it is... if you can imagine the scene from Goodfellas, where Henry walks in the back door of the kitchen, that beautiful tracking shot. You can imagine what that might be like in virtual reality. Actually hearing all of this clanging all around you and feeling like you're really there and able to actually look anywhere. In that case, the realism is as good as the camera technology will allow for. How high of a resolution is it? How good is the playback? Extremely realistic on the film side.

On the real-time rendering side... again, one thing you can bet on in technology is GPU getting better and being able to push more and more polygons at a higher and higher resolution. The demos that we saw in October were running off, I think, next generation Nvidia GPUs. It was on par with what you would have seen in a Hollywood blockbuster, like the first Transformers movie a couple of years ago. Extremely realistic.

I think that will actually be one of the easy parts of virtual reality, in terms of its adoption in the market. Will be the realism of the graphics, if you want to have realism.

I think the other challenge will be the ergonomics at first. It will be hard to convince people to want to buy a headset that looks like a lunchbox on your face. At least for the first couple of generations.

Jen

[00:17:49] That makes sense. You're right, it's not just the visuals. It's also the audio. I'm not nearly as skeptical about audio. Because it seems like we've got that down. Being able to have a whole bunch of small speakers that sound great, that something has been recorded in this amazing way and plays back. It's easier to believe that's already done.

Vlad

[00:18:12] The audio portion is definitely doable. Even on the web, with things like the web audio standard. We can already do high-quality positional audio. But in some cases, it's still not quite enough. There's some interesting techniques for really getting that presence, when it comes to audio, as well, that people are starting to experiment with. I think you'll see some advances happening in that area.

As you say, I think it's a little bit of a simpler problem to solve than the visual piece.

One thing Josh was mentioning earlier — the visual capture piece. I think that part is largely solved, or will be solved shortly. On the actual display piece, the devices that we use to interact with the visuals of these worlds, we're getting the benefit of the explosion in mobile hardware in cell phones. Like the Oculus Rift headset. It's using a regular display from a Samsung Galaxy Note 3. We're already seeing those displays go into 1080p, 4K, potentially even higher in the future. At some point, that visual experience is going to be flawless. You're going to put that on, and you're not going to be able to see the individual pixels. You will just have that perfect presence. W're definitely getting there and I think we'll see it within a few years, probably less than five.

Jen

[00:19:26] It seems like part of it would be... that everybody has to have basically the same hardware, wherever they are. If I think about web audio, what you just said about using the web audio API to do realistic sound that's placed around a person. Easy to do, but impossible, perhaps, to deliver. I just have two stereo crappy speakers from 1995 on my desk. I don't have any sort of surround sound of any kind.

Are you seeing a lot of competition in the hardware space? Like, on the one hand, we'll get amazing equipment because of competition and because of a race to be the best. On the other hand, especially to work on the web, you want everything to be interoperable. This Betamax, Sony, VHS-tape thing. Where you want everybody to have hardware that actually plays with each other. Everybody can have a different headset or device and everybody can see the same kind of things to experience the same kind of experience. Is it like, all competition and everybody's... what's the word? Closed? Or are we going to end up in a world where there will be any kind of interoperability between hardware?

Vlad

[00:20:57] That's really one of the main goals of WebVR. We do want to make sure there is that interoperability. The web is really great at democratizing content, right? You can go to any website, sometimes the fallback isn't as graceful as you might want it. If you're looking at it on a small screen versus a big screen. But it's possible to do that. When we're building out the underpinnings of WebVR, we're very careful to not really tie it to any particular device, to any particular display format or anything like that. We're making sure the content created for WebVR is done in a generic way. The browser — or whatever the piece of hardware that you're interacting with — can then format for that specific piece of hardware.

To give you two examples right now, we have the Oculus Rift hardware. Which is on the highest end of easily available hardware. You can go and buy one of these online right now. That will give you pretty high-quality experience. Good resolution, low latency, all of these good things. But you need to really plug it in to a fairly high-end desktop computer and you need to have a fairly beefy graphics card. All of these things to be able to actually take advantage of it.

On the low end is Google's Cardboard. Which is, literally, a cardboard self-assembly kit that you put together and put your cell phone in. It uses your cell phone orientation sensor — which is already in there — and delivers a low-end VR experience. Something like WebVR can work with both. The same content can be displayed through the cardboard or Oculus Rift. You'll have advantages and disadvantages of both, but we really want the experience to scale depending on the hardware that you're using and not really be limited to either the very high end or bound to the low end.

Jen

[00:22:42] Wow. So what are some projects that you're making? What kinds of experiments are you working on right now?

Josh

[00:22:48] My role on the team is UX designer, in more of a UX capacity and front-end development. One of the challenges we set for ourselves back in October was, what is it actually going to be like to surf the web in virtual reality? If we believe that one of the strengths of the web is being able to seamlessly move to any site from any site. How do we recreate that in VR? Especially because you can't see your keyboard. How on earth do you enter a URL?

We created mozvr.com. We tried to begin to prototype a seamless web browsing experience in virtual reality. Essentially, what we the user saw — because we just changed the site a week ago — they put on their headset, go into virtual reality. They'd see kind of a homepage... which, by the way, for the web designers listening, you have no idea how much fun you're about to have designing web worlds that are actually fully 3D with a genuine sense of scale and placement. It's unbelievable. It's like web designer's architecture. It's the most fun you're ever going to have.

I got to design a homepage, and put links into that world where the user could click by looking, like a look-based cursor. Then tapping a keyboard button, the mouse button or gamepad button to initiate a transition to the next world. You know when you click on a link on a website, the screen blinks white and the next screen draws in, the elements popping in? In virtual reality, we can do interesting things with transitions. If you want to make it seem like you're stepping through a doorway, we could do that.

In our case, we did a transition, kind of a Star Trek teleportation transition, to bring you to the next site seamlessly. No need to take your headset off, no need to install anything. You just hop from one site to the next.

That was the mark one version of these interaction experimentations. Going forward into this year, we want to do more and more of that. One of the limitations of our first iterations was you couldn't actually type a URL in. You could only got to websites where we provided you with this kind of heads up display, a browser as a heads up display, if you want to think of it that way. This year, we want to prototype a much more robust browsing experience that actually lets you go anywhere from anywhere.

We also want to figure out the development patterns. If I'm a developer, what is the equivalent of responsive design for virtual reality? If I'm Wikipedia, I don't necessarily want to make a full VR version of my website. I could, but I probably don't have the resources or inclination to do it. How can I add a little bit of CSS that understands that I'm in a VR context and perhaps positions things in more of a 3D sense? Or maybe increases the text size to compensate for a low resolution of the device. What are the development methodologies that web developers are going to use?

I think this is going to be really, really fun. We've got some pretty neat ideas we're going to be releasing and talking up in the next couple of weeks.

Jen

How much overlap is there? WebGL is a topic that we've talked about on the show before. It's very much supported in... not all of the browsers, of course. But more and more browsers these days. It's something that people can actually ship. Yet, it feels like, in most every day professional web projects, you go to work, you've got this big media website to build, or you're a startup and you're making this app online. I don't see people using WebGL or using any sort of 3D rendering at all. Even when it seems like, "Hey, there's a data visualization. Why don't you do your data visualization in 3D?" Have a controller so people can hold the model of the object and spin it around and play with it.

Where am I going with this question? [Laughs] It feels like there's an overlap and there's a way in which maybe WebGL will pave a bit of a path that you guys will follow. How do you see it?

Vlad

[00:29:41] There are two aspects to getting VR on the web, and GL is definitely a big part of it. As you mentioned, the adoption of WebGL across all of the browsers has been growing steadily. There's a lot of interest now in doing things like you mentioned with WebGL. One of the challenges has been that WebGL requires skills that are not your typical web development skills. It's only been in the last year, maybe two years, that we've had the tools and libraries and frameworks that will allow people who are not pure 3D developers to create WebGL content. We've already seen WebGL visualizations appear in the New York Times online and other places. People are definitely starting to experiment with it. The browser ubiquity, especially on mobile, is really making people take another look at it.

For the VR side, there's two components. The first one is, WebGL content is certainly going to the first that you're going to see. All the experiments that we're doing right now are done using WebGL. Because it gives us full control over that 3D environment, that 3D space. That's both a stopgap and an end result for some things. For example, games on the web using VR will likely be done using WebGL. But we don't want to stop everyone and make sure that we can take the current web — all of the goodness with the DOM and CSS and all of that — and make sure that we find a way that developers who have those skills can also use those to expand into VR.

Just like developers started developing responsive websites that will go from desktop to tablet to cell phone screens, we want to add another aspect. We want to add the ability to make responsive websites that will also do VR. That's going to be something that we'll need to figure out how to do. It's going to depend on, what does the browser for these VR sites look like? What does the user agent look like?

To give you an example, let's say you're looking at a classic website, presented as rectangle on the screen, virtually, in your view. If the text is in VR, maybe it can do some simple things. When you start playing a video, it can just pop the video off to the left. It can keep playing while you're looking at the rest of the content. If it has a navigation sidebar, that can move over to the right, so you would turn your head slightly to look at it. It's accessible and there but it doesn't cover up the main content. We want to give the ability to developers to take small steps towards VR using the same CSS and DOM skills that they already have. Then also with WebGL, give the ability for the full control over the visual experience.

Jen

[00:32:32] That's an amazing thought. They turn their head to see the navigation on the side but otherwise they can turn back and it's gone. That's what people are using the hamburger icon for right now. You have to click and the thing slides into view and you click and it slides out of view. But for it to just be, "Oh, it's over there on the side." [Laughs]

Josh

[00:32:59] I'll give you another one. One of the outstanding things to me, as a designer, again, the resolution of the display is pretty low, so you have to make text really large. One of the solutions to that is, Oculus comes with a positional camera that faces you. It tracks you body's position in 3D space. What you can do with the DK2 is, when you lean in, you'll be leaning in the virtual world as well. That means I can make a little bit of instructional text that's very small but if the user wants to read it, they can physically lean their body towards it in VR. That would like the pinch and zoom of VR, if you want to think of it that way. It really is astounding.

The other remarkable thing is, because you can lean in like that, the parallax effect is extremely, extremely powerful. Let's say I've got two layers and they're superimposed over one another. If I create a one centimeter separation with them — let's say one rem separation — that is immediately apparent. That separation, playing around with z-depth, that's the new drop shadow. [Laughs]

As much as drop shadows are maligned, it's a tool that designers with have in a tool belt to create definition between layers in VR. It's just one of those things as a designer you just have to start playing with. The moment you get your first VR site on screen and you begin to tweak with the variables, there's nothing else like it. I've been doing design for a long time. I'm really excited to see what really talented web designers do with this stuff.

Jen

[00:34:22] What's another user experience puzzle that you've been trying to figure out?

Josh

[00:34:27] I think a major one for us is going to be security and privacy. It's an amazing medium and we can make you believe you are in a world. A lot of us think the potential for trauma will be there, as well. Try to think of the most obnoxious, obscene thing you've ever seen on the web. Imagine that in virtual reality, where you believe you are there. Imagine someone putting something really terrible six inches from your face. That would be bad. The user agent should protect you. The user agent works on your behalf to keep you secure and maintain your privacy. As we enable and open and unbound VR web, how do we give the user tools to protect themselves?

It gets very science fiction very quickly. Already we're playing around with the notion, "What does the browser even look like in virtual reality?" Heads up display probably makes sense. You want it to be away most of the time, then you want to be able to call it forward when you need it. Kind of like a Tony Stark kind of helmet that disappears and reappears. But if you go to a website that is offensive in some way, is there a way to occlude the geometry that's rendered? If they try to put something six inches in front of your face, you've got a buffer zone, a security defense layer that says, "This website is not trusted, therefore it cannot render anything closer to me, my camera than four feet," or six feet or eight feet. It becomes an occlusion zone the user can specify.

Moreover, there's interesting opportunities for anthropomorphization of security. Maybe trackers actually appear as buzzing flies that can be swatted away. Maybe your history — cookies — manifest themselves as a bunch of cans dragging behind you. There are fascinating opportunities for browser designers to play around with how we represent the standards of the web with VR.

If we look at the strength of the browser on mobile, it really has waned. If you think about what a browser is, it is essentially a frame that allows you to launch content. But that's really what the OS does. The browser becomes almost redundant; a frame within a frame. Moreover, there's very little room for that second frame on that tiny screen. If you look at mobile browsers today, they're almost all identical. Firefox distinguishes itself with a swoop in the upper righthand corner. With VR, you've got an entire world to play with. Both as a designer and as a browser vendor. One can imagine there might be a browser aimed at kids, where the manifestation of Chrome was actually a little character. A character that told you about a site being safe or unsafe. You look down to your left and see that character. Or conversely, if you're into VIM and Emacs and Sublime, you might want something that looks more like something out of a novel like Neuromancer or Snow Crash; extremely high-tech and designed for multitasking.

I think one of the subtle implications of VR is that the web may become roaring back in a big way. General purpose browsers may come roaring back, because they'll be easy to create, thanks to these all of these nice, modular, embeddable browsers nowadays. There will be so much creative potential for differentiation.

Jen

[00:37:43] Yeah, there's a lot in what you just said. [Josh laughs] A lot around what a browser is. It is just this boring rectangle frame that we take for granted. We just assume that's what a browser is. Perhaps, even on flat screens, a browser could be something very different. I don't know what. What do we want out of a browser? Bookmarks have been the same for 25 years. History has been the same way for 15 years. The URL bar. Slowly, there's a little bit of evolution in each of these things, but there haven't been any major innovations since the very beginning. And just the idea of the web becoming something... the web itself is so powerful and the browser window seems so... not powerful. Anything that finds a new way to participate on the web, to have a kind of a literal browser. This is the interface for you to get into the web. That's something more than what it's been. It could be very powerful.

Josh

[00:38:57] The way we look at it at Mozilla, the browser used to have a monopoly over your access to the web. By and large. There were exceptions, like FTP clients and email. Especially since the advent of mobile, we've seen the speciation of the browser the diffusion of the jobs we used to hire the browser to do, into the operating system. Now, if I want to search for something in Yosemite OSX, I probably hit command + space. Whereas, I used to fire up a browser tab. If I want to check Twitter, I use my native application for that on my iPhone. I wouldn't actually go to Twitter through the browser client, through mobile Safari. We've seen diffusion of the web and the jobs we hire the web to do, into any number of manifestations — like RSS, push notifications, embeddable browsers.

It's been kind of bad for the amount of time that we spend in browsers. So, probably not really good for browser vendors like Mozilla. But it's been really good for the web. The web is now everywhere. It is completely diffused everywhere. It's really been commoditized. We seek to commoditize the web in virtual reality to the same degree as it's been commoditized on desktop and mobile.

No matter what kind of VR application you're making — whether you're a web dev or native dev — no matter what headset you're running on — whether it's Sony, Oculus, Microsoft, Google, whomever — you can easily integrate the web. You've got the APIs to do it, embeddable browsers are rife, and there's reliable, agreed upon standards in the community for how you can create content and create this stuff.

We figure we've got about a year. That's when we think Oculus is going to ship. We'd be very, very happy if in a year, these embeddable VR-enabled browsers and these standards were out there and ready to be used by developers.

Jen

[00:40:49] Nice. You spoke before about mozvr.com. What kinds of things are here? What can people look at now?

Josh

[00:41:02] We started off with... I just had to learn the toolset. I hadn't done a lot of WebGL myself. I've been a designer on Firefox OS previously. I've done 3D in the past. But I had to learn WebGL, the team had to learn WebGL. mozvr.com was where we began to experiment and create WebVR experiments.

As of last week, we've updated the site to be more of a resource for developers. Developers can go there, see the demos that we've created — these WebVR, WebGL demos — but they can also get boilerplate code from our GitHub repo. So if you're listening to this and you want to create the VR scene, you can go to github.com/mozvr and you'll see a bunch of repos for everything that we do. We publish everything. There's a VR web example repo there. It's basically Three.js and heavily, heavily commented with everything you need to create your first VR world with a spinning cube and VR_enabled scene.

The rest of mozvr.com, we're going to try to make it a resource site with tutorials and blog posts. Being Mozilla, one of the wonderful things is, we can share everything, and we win when we share everything. We want to start to do that as much as we possibly can.

We've got some pretty neat tools in the pipeline. Whereby a developer can lay out a 2D HTML and CSS site, fire it through a build script, and have a WebGL, 3D virtual reality scene spit out on the other side. So a developer like myself can use the skills that I've cultivated for a long time — HTML and CSS — to create these really rich VR experiences.

There's a whole bunch of stuff like that, that's we're going to be releasing over the next six months and we'll be publishing it through mozvr.com.

Jen

[00:42:38] Nice. I assume if someone out there does go to your site and create something, they should ping you somewhere on Twitter or someplace and let you know and let you check out what they've done.

Josh

[00:42:51] If they're inclined, totally. But hopefully this gets so big that just doesn't scale. [Jen laughs] That would be a good problem to have, you know?

Jen

[00:42:58] Well, if they do that this winter, or this spring into summer, they should show you what they've built.

Vlad

[00:43:04] Absolutely. And if they have any questions, as well, we're pretty accessible on Twitter. I should say here that we've been really fortunate to work with some amazing teams in the early WebVR community. There's one team called eleVR here in San Francisco who published a ton of stuff, open source.

They made a WebGL VR video player that we used to publish a VR documentary on mozvr.com called The Polar Sea. You go to this website, you hit the VR button, and you're flying on a helicopter over the Arctic. This documentary team sent us an mp4. We dropped it into eleVR's open source WebGL video player and that was all we needed to do.

We're pretty excited about the distribution of VR video through the web, in particular. You're probably not going to be inclined to install a music video on your Oculus headset a year from now. Or on your Sony headset a year from now. But with YouTube, and sites like YouTube, you want to be able to consumer a piece of footage and move on to the next thing. That kind of consumption-without-obligation model is where the web excels. We just want to make it as easy as possible for people who are creating VR video content using these new cameras, to publish to the web. Then anyone on any headset on any platform can watch that content easily.

Jen

[00:44:20] Yeah. There's a big use case I could see. So Firefox nightly supports WebVR, is that right? Will you explain what the deal is with that?

Vlad

[00:44:33] We have the needed additional APIs for WebVR in Firefox nightly. Also in the Firefox developer edition. Although, the development on WebVR is pretty rapid, generally we suggest anyone experimenting with it use our nightly builds. That includes the core APIs, but to actually use the Oculus headset requires the installation of a separate add-on. All of this is described on the mozvr.com webpage. There's installation instructions there.

One of the things that we're working on is adding support for additional devices, especially for mobile. The Google cardboard device that I mentioned earlier. Which should require no installation. Especially for things like VR video, we want to get it to the point where you can install Firefox for Android, on an Android device, or use a Firefox OS device, put the device inside a Cardboard unit, and be able to click on a VR video link and experience it, even on the go on your device. It's in Firefox nightly and also the Firefox developer editions. Oculus Rift, like I mentioned, requires a small add-on install. But that's something we're hoping will go away in the future.

Jen

[00:45:42] I'm on the site now. It's actually really quite clear, if people want to go and check out mozvr.com. Click on downloads. You've got it really good and clear. It's nice. You don't always see things that are this clear. "You need the flunky-schinka-flunga. Of course everybody already knows about it." [Laughs]

Vlad

[00:46:02] You're making Josh very, very happy. Because he's been on my case to make the install as simple as possible. So that's really good to hear. [Josh laughs]

Josh

[00:46:11] Thank you, Jen. [Laughs]

Jen

[00:46:12] There are people driving in their cars right now going, "Oh my god! I want to go home and take my Oculus headset and get on the web!" And then they're like, "Wait, what did you say?" Are they going to pull over in their car and write down everything? I mean, a) you can go to the show notes, but b) you can just go to mozvr.com and, yes, there are very clear directions and exactly what you need.

Josh

[00:46:30] If you go to mozvr.com on a mobile device, I apologize. My CSS-fu is rusty when it comes to responsive design. It's a bit busted right now, but we'll fix that.

Jen

[00:46:40] You'll fix it. [Josh laughs] Oh, yeah. It's not that busted. Just a little bit. You guys have screenshots of how to set up the monitors and everything. Very nice.

Josh

[00:46:51] We just published a post last week on a technique we've been using internally to go from Illustrator to a WebVR mockup. We're really big believers in that designers have the toolsets — things like Illustrator, HTML, and CSS. You really should be able to start creating VR worlds with the toolsets that you have. You shouldn't have to learn WebGL, as powerful as it is, and I do recommend you look at it. You should be able to save a layout from Illustrator and dump it into a WebVR scene.

One technique we've been using that we really like a lot is, I compose my mockups on a canvas that's 360 centimeters wide by 90 centimeters tall in Illustrator. What's neat about that is, that canvas, the texture that I save out, gets mapped onto a cylinder on WebVR. By making it 360° wide, I know that each centimeter on that layout is going to equal one degree of rotation on the VR scene. So I can actually have a really predictable system for doing my layout.

For web designers, one of the fascinating things they're going to have to get their heads around is that, in VR, you have to work with physical measurements, and scale really matters. If you set something to be one centimeter, it looks a heck of a lot different than something at 1,000 centimeters. Conversely, if you want to have text as big as a mountain, you can do that, and the user will perceive it. I cannot overstate — and I know it sounds a bit crazy — the degree of power in how awe-inspiring it is for somebody to be doing web design with that kind of ability. There's really nothing else like it.

Jen

[00:48:21] Yeah, I can barely wrap my head around what you just said. [Josh laughs] Like, centimeters? Count? How can a centimeter count on the web?

Josh

[00:48:29] The core measurement of VR is how far your eyes are apart. It measure it in meters. That is the measurement that's reported to the WebGL scene. Vlad, feel free to correct me at any point here. As a result, when you're defining your world in virtual reality, you're defining it in meters, like the eye separation is defined. So physical measurements are the core of everything. Because the eyes are seeing different images, you're getting true stereo, true sense of depth. You can really perceive how far away things are.

We did one version of that homepage we mentioned earlier. I had set a distance of 1,000 meters — 1,000 is an easy number to work with — and it was like being at the top of a building, looking at each one of the links I built as a different building, 100 blocks away. I divided the scale by 10, and all of a sudden, each link was a cupcake floating about a foot away from my face. As I leaned by head a little bit, they would begin to have this parallax effect.

Again, it's so hard to describe these things. [Josh and Vlad laugh] You have to try to really wrap you head around it. But if you've got a DK2, I highly encourage you to start playing. If you've got a friend who has one of these headsets, go hunt them down. Oculus said there's about 100,000 of these. When I say DK2, by the way, it's developer kit. You can buy these developer kits from Oculus. They're not commercially ready, but you can go and buy them for about $400. Or you can get the Samsung Gear VR, which is a holder, essentially, for a Galaxy Note 4 that you can slip in. That's a very affordable, really compelling VR headset that's available now. Not compatible with our VR yet, but we're working on that.

But don't take my word for it. Go and hack and start to play. It's pretty darn fun.

Vlad

[00:50:22] And Jen, you actually mentioned one interesting thing, when we were talking about centimeters and meters and all of that. That's one of the challenges that we have now, trying to figure out how to marry CSS to VR. Because CSS has all of these units. You can specify things in inches and centimeters, but they all sort of end up as pixels on the screen. Maybe you have 96 dots per inch, and that's how you decide how many pixels you have, but it's easy to zoom in and out.

Whereas, when you're talking about VR, if you're trying to lay out a VR scene using CSS, all of a sudden, it does make sense to talk about, "This <div> element is actually 10 meters by 10 meters and it is 100 meters away in this direction." That actually has meaning. That's one of the challenges and we're figuring out, "How do we actually allow you to create these things of scenes using CSS, using the existing web tools?" Also, how do the web engines interpret that? How do engines that have only had to deal with pixels, all of a sudden have to deal with meters and centimeters and millimeters and all of these things that have some kind of meaning beyond translated to pixels?

Jen

[00:51:33] Yeah, I wonder. It's one of things where, nothing was digital, and suddenly things were becoming digital. The first round for many of us was print design. It wasn't really Illustrator and Photoshop, it was the software that were the predecessors to the tools that we use today. Of course you used centimeters and meters and inches and feet. But when the web came along, all of those people jumped on the web and it was years and years and years of saying, "Don't think about this as inches. Don't think about this. It's pixels, it's pixels, it's pixels." But, yeah. CSS has those units in it. But they became... I don't even know. I'm going to start saying things that are probably technically wrong. Our orientations this whole time have been... if I opened up somebody's CSS and saw centimeters, I'd be like, "You're doing it wrong. This should be written in pixels." Or ems, or rem, or viewport units. There's options.

But it feels like one of those old technologies that was misused for so long, that all of the browsers probably have this weird way of interpreting it that's just, "Uh, yeah, that didn't mean anything, so we made it mean this thing a hundred million years ago and now we're stuck with that because we can't change it. Maybe there's somebody out there who's using it and we'd break their website if we changed the way this is getting interpreted."

Perhaps that's how it is. Now you're coming along being like, "No. We really mean centimeters. We actually mean it. We're building a virtual space that is physical. It's a physical, virtual space. We're going to measure it in the metric system. Please let us use centimeters properly, people who wrote the web in 1996."

Vlad

[00:53:26] We sort of ran into that same problem when high DPI display started coming out. Because all those units — centimeters, meters, inches, everything — it was all based on one inch is 96 pixels. That's where everything tends to fall on the web. But then all of a sudden you've got somebody's high resolution retina display, where one inch is actually 200 pixels and everything looks really, really tiny. Then all of the browsers have to do magic under the hood to map that 96 DPI into the actual, real resolution. Then you had all of the problems with, "What if a webpage wants to display a high resolution photo? How is it specified?" No, it's the actual pixel width. We're just extending all of those problems into VR.

One thing that I didn't realize until I started working on this, CSS has inches, points, centimeters, millimeters. There is no CSS unit for meters. Because it doesn't really make sense. A meter, when you take the 96 DPI, would be huge. It would be an enormous amount of pixels. But you want the meter when you're talking about VR> Little things like that crop up.

Jen

[00:54:35] Yeah, maybe you need new units to appear in CSS. Maybe you just go with meter and ask people not use centimeter. Or go with <vr-cm>, some sort of new unit to get around the problem of not being able to use the old unit properly.

Josh

[00:54:51] It's hard to work in meters when you're dealing with .01. It's just very difficult. We tend to work in centimeters in HTML. For example, my setup right now in HTML, I define the body size as 10 pixels and then I set the body width to 360rem. So I can work in that 360 system I was alluding to before. Then we do a conversion once we import into WebGL via this tool — that as mentioned earlier we're going to be releasing soon — which does a final conversion into more of a meters-friendly scale. So the designer shouldn't have to worry about it. They should be able to work in the units they want to work in, and have the conversion taking care of it at the end. That works pretty well.

Jen

[00:55:33] Nice. Long time listeners of this show will know, Nellie McKesson was on episode 52. She's been doing a lot of work with CSS and laying out books for years now. When she writes CSS, she's using inches, she actually means it. She means inches. I should have the three of you connect so you can figure out if any of your frustrations are at all similar and if CSS needs to change in some way to support both use cases, both virtual reality and actually using CSS to make physical print.

Josh

[00:56:18] One of the really nice things about working at Mozilla is, if I have a frustration with CSS, as some of us encounter from time to time, I can look down the hallway and throw something at the guy who wrote the spec. [Jen laughs] It's kind of a nice thing to have.

If anyone who is into that stuff is listening, I will make the same plea to them that I've been discussing internally. It would be neat to be able to define latitude and longitude coordinates in CSS. As we go into an IoT augmented reality, mixed reality future where you can walk around with something like Google Glass, but much less horrifying than Google Glass, and much more ergonomically pleasing. It's going to be pretty neat to be able to define a website that lives out there, in a physical location in the world. It seems like one easy way of doing that might be to define a <div> as existing at a specific coordinate on the planet, and position itself relative to that coordinate space.

Again, it all sounds pretty science fiction, but this stuff is coming pretty fast, so we want to start thinking about these things.

Jen

[00:57:14] Speaking of Google Glass, it feels like up until about now, all of our thinking about watches and eyeglasses — Google Glass and Apple Watch and other watches — has been around imagining a little tiny screen .A little, tiny flat screen that's 200 pixels square or something. How are we ever going to jam a website into that tiny, tiny space? But now, listening to you all, it makes me think, maybe the browser on the watch is actually, as your hand is moving, it's this little loop, a little portal to look at something that's in virtual reality. The actual site stays still, and as you're moving your arm around, you're able to see that part of a webpage.

Have you thought about watches and glasses and how what you're doing might change what other people might be doing on those things for the web?

Josh

[00:58:14] At a high level, yeah. I think it's all part and parcel of this trend of computing getting smaller, faster, and cheaper. We joke that the definition of mobile is changing. Mobile has been a smartphone for the last seven years but it's kind of exploding like a piñata. Just like the browser got unbundled, so is a mobile phone getting unbundled into watches, into these ubiquitous iBeacon, HomeKit-powered sensors that live everywhere and connect their experience, and the cloud services that integrate into these individual components.

Leaving aside VR or AR or any particular acronym, that trend seems to be pretty obvious. It's pretty interesting, if you fast forward a couple of years, to think about... let's say you've got an Apple Watch. Let's say that Apple Watch is able to pick up muscle movements. It can already pick up accelerometer data. Maybe a couple of versions down the line, it can actually determine which finger is being moved and to which degree. There are devices out there that already do this, Myo ArmBand for example.

You can imagine that an Apple Watch might communicate with some future Apple headset to enable your watch to be the input device or to recognize your arm movement and recreate your arm in VR. You can imagine that, maybe Safari a couple of versions down the line is actually on your wrist. In virtual reality, if you want to call up the browser interface, you just look at your wrist. You move your real world wrist to look at it, and it pops up, super-imposed.

Because we're talking about the intersection of multiple devices — from wearables to IoT to headsets — to even, eventually, a brainwave-pickup device and input devices like that. It's very difficult to make accurate predictions about what specifically is going to come down to line. Which speaks to Vlad's point earlier. It's so important that those of us working on the web who really care about interoperability and backwards compatibility and progressive enhancement design some flexibility into the system. No matter what comes down the pipeline, we can integrate it effectively into the web.

I keep saying it, but this stuff, science fiction is coming pretty darn fast. A year and a half, two years from now will look pretty weird. [Laughs] It's going to be pretty exciting.

Jen

[01:00:31] Microsoft just announced the HoloLens. A kind of MR technology. What do you think about that?

Vlad

[01:00:43] What I like about their approach is that it looks like it's going to be home-based. They're not trying to simultaneously take on all of the immense challenges of doing mixed reality headsets in computing while also trying to make it mobile. If you just throw mobile out the window, you don't try to replace the phone, you don't try to make it something that looks good when you're walking down the street, maybe that's a good starting point. Maybe you can go after enterprise or education or home-use, where the Wi-Fi is particularly strong and you've got great connectivity. Then you can grow it from there as the headsets become smaller and smaller and smaller.

I heard some mixed reviews, but most reviews seemed to say that this is magic, this is sci-fi, and, "Oh my god, I can't believe this is actually here." I think the one — and this is probably outside the scope of this podcast — I think some folks are right, there's a bit of a question mark there about the timing of the release and how real it is and their ability to really execute on the vision.

I will say that there's a lot of speculation about another company out there who appears to be doing the same thing, called Magic Leap. This is a company that Google recently led a round of investment the tune of over $500 million into. Based out of Florida. Based on trademark and patent filings, appears to be next generation mixed reality headset. All of their trademarks are all about the metaverse. They just feel very Neal Stephenson, Snow Crash, they're quite obviously trying to build a metaverse kind of device. Think about a transparent display that achieves effects and results similar to the HoloLens, but is perhaps more mobile; you can wear it around with you. Google has led a round of investment for them for over $500 million dollars.

Sundar, from Google, is on the board of Magic Leap. All of the stuff you've heard about Google Glass being contracted and moved under Tony Fadell of Nest in Google. This is totally speculation as a VR fanboy, I think we're probably going to see something like Magic Leap be your consumer Google Glass in a year or two from now. Totally speculation. I don't know anything. Just speculating on what's publicly known.

That's just Google. There's still Sony and Facebook. There's now Microsoft. One has to believe that, insofar as Apple is a computing company, if wearable displays and wearables are the future of mobile computing, that one would expect Apple to make a play in this space in the future. Frankly, I can think of few companies better equipped to actually do that. I would love to see what Jony Ive does with a headset. I'd love to walk into an Apple retail store and actually try it out.

Jen

[01:03:21] All of this makes me really excited about what you folks are doing. It feels like, if this comes to be, where computer devices become what you're talking about — this virtual, interactive thing — I want the web there. We want the web to be right there, in the midst of all of it. We don't want to have these separate walled gardens where you either get into this particular brand, this set of apps, this brand, that set of apps, all competing with each other with no option. Then the web kind of goes away. We want the web to be part of it.

Josh

[01:04:01] Yeah. I think what will probably happen is, you'll continue to see walled gardens, you'll continue to see app stores. Because, frankly, people like them and are familiar with that model and they work pretty well and there are a lot of incentives amongst guys like Facebook, Google, and Apple to actually maintain that kind of model.

What we want to see is for the web to live in a symbiotic relationship with those models. Perhaps, to do more. To be in a stronger relationship. The web isn't something you add on to 5% of your application. Maybe you can do 30%. Maybe you can do 90%, depending on what you're building.

The way we think we get from here to there is starting by the kinds of things that Vlad and Brandon did last year. Finding these Judo-move APIs where you can take all of the existing power of the web and add a couple of APIs on top of it to unleash some new capability. That's the kind of approach that we believe in.

Jen

[01:04:51] Is there anything else? We're about out of time, but is there anything else that you want people to know about? About what you're working on?

Josh

[01:04:59] If there's folks out there who want to start playing with this, don't hesitate to reach out to us. Because the community is still pretty small right now, we don't mind hopping on calls with teams who are interested in getting started. For example, if there's a team at a company like Wikipedia who's interested in looking at this, we'll typically hop onto a call and do an orientation session to help them to understand the technology and how it works.

Vlad can speak to this better than I can, but we're going to be expanding support to mobile devices soon. It won't just be a desktop thing. Then we'll be publishing a lot more through mozvr.com, including this tool we're developing where you can go from HTML to WebGL with just a build script that runs. It's going to be a pretty crazy year, I think. In the VR community, we're all pretty pumped. We're excited.

Jen

[01:05:50] People can follow you on Twitter. What are your Twitter handles?

Vlad

I'm @vvuk on Twitter.

Josh

And I'm just @joshcarpenter on Twitter.

Jen

Do you have a Twitter account yet for the MozVR project?

Josh

No, people just throw things to hashtag #mozvr and that works pretty well. The nice thing about MozVR as a name is that it's very unique. Nothing else gets confused with it. So that works pretty well.

Jen

Nice. That's cool. People should definitely check out your blog and follow what you're doing. Alright. Thanks so much for being on the show. This is fascinating.

Josh

Thank you for having us.

Vlad

Yeah, definitely, thank you.

Jen

We'll grab all these links and put them into the show notes for this show, which people can find at thewebahead.net/96. Where we'll also have a transcript of the show. So, thanks. Thanks for listening.

Josh

Fantastic. Thanks so much Jen.

Vlad

Thank you Jen.