Transcript

Jen

This is The Web Ahead, a weekly conversation about changing technologies and the future of the web. I'm Jen Simmons and this is episode 99. I first want to say thank you so much to our sponsors today, Squarespace and Code School. I'll talk more about them later in the show. I also want to say thanks to Pantheon for powering the The Web Ahead website. And to CacheFly for being an awesome CDN, delivering all of the audio files to you.

Before we get started today, I should tell you — I keep realizing I'm not promoting myself enough — I am speaking at quite a few conferences this year. The Responsive Web Design Summit is coming up very soon. I might be talking right now, as you're listening to this podcast. If you're interested, they record everything. It's an online conference. All the videos are available afterwards, if you want to pay to access them… It's never too late to attend the Responsive Web Design Summit. [Use the code JEN for 20% off any ticket.]

I'll be at An Event Apart Seattle in March and An Event Apart San Diego in June. [Use discount code AEASIM for $100 off any two- or three-day AEA event in 2015.]

I'll also be keynoting the UIE UX Immersion Mobile Conference, which is interesting. It's two solid days of hardcore workshops on how to do this stuff. Especially around user experience design in the mobile context. There's one day of talks, and I'm one of the keynotes. [Use the code UXIMSPK to get $300 off the full conference price.]

There will be others — there are other conferences coming up throughout the year. Some that are in the process of being booked. Probably at least another three or four. You can always go to jensimmons.com and I'll update my list and make sure everything is there will all of the links and coupon codes [or see above] to get discounts and such. I'm doing three different talks this year, but most of the time I'm doing one talk I'm pretty excited about, around layout and layout innovations. Showing examples of CSS Shapes, CSS Exclusions and Flexbox and other ways to think about layout. Come meet me in person! Come introduce yourself, hang out, we'll have dinner.

Today we're going to talk about responsive images. The last couple of shows — and the one next week — have been big-picture shows. What is this thing that we call the web? How are we archiving it? How are we handling advertisements and privacy and liberty? Big-picture stuff.

Today's show is going to be none of that. [Laughs] Today's show is going to be really specific, practical, current-day, how-to, and why, around responsive images. My guest today is Jason Grigsby. Hi Jason.

Jason

Hi Jen. I just want to say that we are going to be talking about big pictures because they are, in fact, the big problem here. [Both laugh] Excuse the pun.

Jen

[00:03:20] Yes, big pictures! [Laughs] And how to make them little, so they fit on your small devices and your tiny, slow internet connections.

Jason

Exactly.

Jen

You've been on the show twice before. You're going to be in the three times club.

Jason

It's an exclusive club. I'm very excited that you've made it to 99 episodes, and I'm on episode 99. I got a little sneak peak at who you've got for 100 and that sounds awesome. But 99 is a close second. I'll take it.

Jen

It's a pretty good number.

Jason

Yes, exactly.

Jen

I started loading this last one into the new website and I realized, "Oh, look at this! I didn't make space for three digits." [Both laugh] I've got to redo my CSS before I can publish this show.

Jason

You've got a Y2K problem.

Jen

Yeah. It's just smashing into the title, so I've just got to make a little bit more space. But you were on episode 16 to talk about mobile and mobile phones and how the web is going to be changed by mobile phones. Then you came on episode 22 to talk to television sets.

Jason

[00:04:36] That's right. I'm still waiting for Apple to release a TV. Then all of the hours I've spent inside consumer electronics stores, testing browsers on TVs, will have been worth it. But I wait. And watches arrive instead. [Laughs]

Jen

Yeah, it's one of those many things that feels like it could be big, and then it's just not. At least not yet.

Jason

Images have been another a constant over the last few years. We've finally reached a point where it's time for people to pay attention in a way that they probably weren't in the past.

Jen

In episode 25, Matt Marquis came on to talk about responsive images. Which was two and a half years ago. On the page for this episode, we'll have a link to episode 25. If people are interested and want to know more after listening to this episode, maybe go listen to episode 25.

Although, some of what was said two a half years ago might be a little bit out of date. A lot of it was big picture, rah-rah, cheerleading, "We need a solution, this is what we think we're going to do. This is how you can help get involved." The responsive images community group — along with folks who write standards and browser makers — everybody's been working the last several years to get to a place where all the decisions have basically been made.

Browsers are now implementing all of these technologies and now is the time to start using responsive image techniques and solutions on your production websites to solve the problems that we have when we want huge images to go on websites and they're really big.

Jason

[00:06:28] Correct. Last fall, I'd been doing a lot of talks where one of the pieces of advice that I'd given people was, as they were implementing responsive image solutions on their site, that they should really centralize their markup. Centralize in one place the functions that would output the markup for responsive images. Because this stuff may change in the future.

I started thinking about this. We've got browsers who are shipping this. Is this still the case? Do we still need to hedge our bets in that way? I asked the browser makers and people have been involved in standards for quite some time. They said, "No." It still may make sense. It still may be a good strategy for organizations to centralize. I think it makes a lot of sense for sites to have some sort of central service for resizing images. The whole idea of hand-cutting images was probably something from a bygone era that we're all going to move past.

It's no longer necessary as a risk mitigation strategy. Right now we've got responsive images. The standard is supported in Chrome and Opera. It's coming to Firefox 38, which has a release date of May 12th or something like that. I'm heading up to Seattle tomorrow to meet with folks from the Microsoft team and talk to them about Spartan and it's support of responsive images. Yoav is working on a port to Webkit of the Blink implementation of responsive images. Webkit will have responsive image support before too long. Whether or not that's something Apple ships in Safari is, of course, an unknown. Because they don't tell us what they're going to do.

Jen

Right. It will just show up.

Jason

Yeah. But across the board, we've got to the point where either browsers support this syntax or you can support it using Picturefill and accomplish the same thing. I really do think 2015 is the year. Before now, it was a small subset of people in the responsive images community and browser makers and standards people, all spending a lot of time thinking about how to do this, what the standard should be, and where the problems lie. While responsive images are a big issue for the web in general, it really was a small subset of people who were impacted by the work.

Now it's going to be everybody. Anyone building sites is going to have to spend a little bit to time learning time and figuring out what makes sense for their site. What makes sense on a site-by-site or image-by-image or page-by-page basis is going to differ.

Jen

[00:09:56] It sounds like you can ship it today. It works in half the browsers and by the end of the year it will work in all the browsers. Frequently when we're building a big website, it takes you six months to launch it, anyway. So start working on this.

Jason

Picturefill is a polyfill that will basically replicate what the syntax will be. You can build stuff in a standard manner. For the browsers that don't understand it, Picturefill will solve that and help you not have to worry about coding around the fact that the browsers don't understand it. As far as whether or not the other browsers will support them by the end of the year...

Jen

It's a guess.

Jason

Yeah.

Jen

But it's a guess that sounds like... eh, probably.

Jason

Yeah. I hope so. I hope you're right. I found this email from Tim Kadlec that he had written back in 2012, saying, "Hey, I'm writing a chapter on responsive media. I'm looking around, I was hoping that somebody smarter would have solved this by now." [Laughs] In 2012. The first time I was writing about images as an issue for responsive designs was 2010. The year that I did the multi-part series on responsive images was in 2011. I've come to expect things to take longer than I would have otherwise.

Jen

That's true. [Laughs] For anyone who's 10 minutes in and is like, "I'm not sure what they're talking about." We've had the <img> tag, which you write in HTML, which is short for image. Then src and you put the equals and the quote and you put the link to the image and that makes your image show up on a webpage. That's been around for 20 years. That's awesome. Why in the world do we need a new thing today? Why are we changing that?

Jason

[00:12:10] Well, like everything else, the iPhone changed everything. [Laughs] That is a cliche, right? What we're dealing with is device diversity. If we're building sites that support everything from an iMac retina display all the way down to feature phone display with single density, we're talking about a wide range of screen sizes. Oftentimes, a wide range of image sizes.

If you take an image that might be optimized for that large screen at high density, it could be 500k. That same image sized appropriately for a feature phone might be 20k. That's a significant difference. A lot of what has been done in responsive designs is to have the browser resize images. Within a certain threshold, that makes sense. If you've got an image that's 10% or 20% larger than what it needs to be in the page, that may be sufficiently fine to resize down the way that it is as that particular viewport size. But it doesn't make sense to take something that's 500k and design for retina display and shrink it all the way down to a small screen.

When we're talking about responsive images, we're talking about is ways to provide the browser with multiple source files that are sized appropriately for however large or however small that particular image is going to be used in a design. That's the problem that we're trying to solve. Once you start looking at it, you realize there are a lot of different ways the images are used. Whether we're talking about display density, or the size of the element in a responsive design, there's a bunch of different variations on that theme. Overall, responsive images is about providing multiple image sources so the browser can download something that makes sense based on the image that the image is being used in the page.

Jen

Right. I don't want to repeat what we talked about a couple of years ago, but I do remember a lot of discussion. People saying, "Why don't we just use JavaScript? JavaScript can detect the size of the screen and tell the browser which one to download." It seems like all of those ideas were just bad — or maybe I should say insufficient. Because what you really need is the original, most rudimentary call to the image. To have a syntax that says, "Actually there's not one image here, there's three options for images. Which one do you want?" Get the one that's appropriate for you, whoever you are. Depending on what kind of device, what kind of screen, what kind of size you are. That needed to be in the HTML. That's what we've got now. It's different HTML that you're writing using <picture> or srcset, which we should talk about. Instead of using... or maybe it's better to say in addition to using the <img> element, you're using a <picture> element or you're using one of these other things.

Jason

Correct. On the point about JavaScript or CSS. The reason why we needed what we commonly refer to as native responsive images is because the browser has this function called the lookahead pre-parser. Someone corrected me recently and said it's both that and the regular HTML parser. But frankly, the inner machinations of the browser's engine is a little bit above my pay grade.

What's important to know is that, if you were to open a timeline on a site and start watching the assets load and the events that are firing. First the HTML downloads. The browser starts parsing the HTML once it's got it all. It starts downloading the JavaScript and CSS. Then immediately it starts requesting images. Before the JavaScript and CSS is fully downloaded, before it's been parsed, before it's been calculated, and before layout has been evaluated. It's grabbing as many assets as it can.

This is what the lookahead pre-parser has done. When Google implemented it, they found it increased the speed of pages 20%. IE was actually the first to do it, they just never published the numbers. They just said, "This made a huge improvement." Firefox found that it was a 19% improvement in page speed. Steve Souders talks about it as being the most important improvement to web browsers of the last decade.

Jen

Wow.

Jason

[00:17:35] You've got this natural conflict between the browser wanting to download things as quickly as it can — even before it knows what the layout of it is going to be, before it's finished building the DOM, before it's done any of that work. CSS or JavaScript evaluation. You've got responsive images, where, in an ideal world, we would just wait until the layout is there to download it.

When we think about JavaScript and CSS solutions, we're thinking we could solve this problem because we could put it in JavaScript or CSS because they know the size of the viewport and the size of the element on the page. By doing that, we're taking this huge boon to the performance of webpages and removing it.

It's this conflict between somebody who wants to go on a vacation and have everything planned out ahead of time, and somebody who just wants to roll into a city and take things as they come. [Both laugh] We've been negotiating between those two people and trying to find solutions. When people look at the solution and say, "This is too complex. We could just solve it this way or that way." It's because they're not solving for that problem and they're getting rid of something that is such a big boon for browsers.

That's why we can't solve it with CSS and JavaScript. Fortunately, we don't have to. That's why we have the responsive images specification. We've got the <picture> element. We've got srcset, we've got sizes, we've got type. We've got a bunch of tools at our disposal.

Jen

Let's talk about <picture> and srcset. What they are. Clearly, people should go look up code examples so they can visually see them, cut and paste them, copy them, or whatever they want to do.

Jason

In order to talk about <picture> and srcset, I want to start with the <img> element itself. Because I feel like that's the place to begin. The <img> element is required. It is the basis by which srcset or <picture> work. The way that I like to think about it, the <img> element is like a box unto which the rules in <picture> or the things in srcset get applied. If you instrument this, if you use JavaScript to look at the current src of the <img> element in the page, what you will see is that as the viewport changes and the different rules are applied, that the src of that <img> element changes.

Eric Portis talks about this being like progressive enhancement on the <img> element. Anything you've written that expected <img> to manipulate an image — if you've written JavaScript or something like that — will continue to work. As will all the things that browsers have written that are based on the <img> element. They will continue to work. Because the <img> element is still our toehold for all the reponsive image solutions.

Once you come to the realization that you need something more than the <img> element... and sometimes you don't. You can solve some responsive image solutions with just an <img> element. If the difference between the largest use of an image on the page and the smallest use of that image is small enough, you may not need to provide different sources. Or maybe you're using something that's an SVG. You could, in theory, assuming the browsers you're looking at support SVG, you could use a single <img> element with a src.

Once you decide, "Ok, I need to support different devices," say we want to support different displays or image sizes, then you start adding things to the <img> element. The first thing that makes sense to add is srcset. srcset has two different ways it can work.

The first way is what we refer to as x descriptors. It means there are x's in the attributes. You've got srcset as an attribute on an <img> element, and you've got, say, img1.png 1x. Then a comma — it's a comma separated list — and img2.png 2x. Those are indicators of the display density. The browser will pick the correct image for the display density. This is very similar to what was done on iOS inside native applications. As a matter of fact, when retina displays came out, this was when that syntax was first proposed. For those people who jump back and forth between native and web development, that will seem very familiar. That's the first piece.

The second one is called width descriptors. Instead of having 1x or 2x, you're declaring the width of the image. You might say 240w for the first image, then maybe the second image the 600w. Again, you have a comma separated list of images with those widths attached. You're providing the browser with a series of image sources it can pick from. You're not dictating which image it has to use. You're saying, "Here's a list you can choose from. You're likely in a better position to know which of these images makes the most sense." Because we don't know. Browsers are better suited to know the network conditions. Hopefully, eventually, they'll provide user options or do things differently based on zooming. There's also been talk from browsers about the fact that if an image is less than 10% smaller than the size that it's used in the page, that it may make more sense to make it slightly bigger and it will still look good. Then it makes sense to download the next largest image. That's width descriptors.

Jen

This is where it can get confusing. It could be the width of the viewport. We're used to so many different widths. A lot of the code that a front-end developer might write has to do with the width or height of the viewport. A lot of the code has to do with the width of that particular column or box or section of the webpage. You have a 200 pixel wide sidebar column, you're thinking about 200 pixels. Or it's 40%, but it's only 40% between this breakpoint and this breakpoint. It might be at the smallest 100 and at the largest 250. But there's a number there. It's a width. And sometimes, if you're using something like multi-column layout, you're saying, "Hey, I want you to try to be 200 pixels wide. But you make it up. Sometimes you're going to be 180, sometimes you're going to be 150, sometimes you're going to be 250, sometimes you might be 300." You, browser, make it up. You're going to jump from having two columns to three columns to four columns. I don't need to write a bunch of media queries. I don't need to do a bunch of math. I just want you to try to go for 200 pixels wide. You figure it out. I know you'll probably hardly ever be at 200, but you'll be close enough to 200. That's the goal.

You know what I know? Sometimes you're working in percents and it may or may not turn out to be something. But sometimes you actually are saying, "This is an icon. It's a photo of a person. I want it to always be 100 pixels square, always." My question is, which one is this?

Jason

[00:26:56] Actually it's none of those. [Jen laughs] If you were to open up this image in Photoshop — the actual file — because you're providing a list of files. File1, file2, file3, file4. You open up file1 in Photoshop. You go to get info about this image. It tells you it's 333 pixels wide. That's what you need to put in there: 333w.

Jen

So you're telling the browser, "These are the names of the files I have. These are the widths of these files. Do with them what you will, or we'll talk about this later. In a little bit, as you load, you'll be told what to do. Here they are."

Jason

To be a little bit more specific, it's actually URLs. Paths to the images. But I'm not going to read slashes on that. [Both laugh]

Jen

So here's a file, here's the width of the file. Here's another file, here's the width of that file. In pixels.

Jason

Correct, in pixels, yes.

Jen

In real pixels. [Laughs]

Jason

Correct. Real pixels. Not virtual pixels, real pixels.

Jen

Right.

Jason

So that is the first piece of the puzzle. Now we've got these two different versions of srcset. One of which has just display density, which is simple, but a limited use case. The second one is, you've provided the browser with different image files and the widths of the images in those files. That works. That would be enough information for the browser to make smart decisions about which file to download, based on the size of the viewport. Because that's all the browser knows when it starts requesting images. If we think back to the pre-parser I was talking about, the only thing the browser knows when it starts downloading images is the size of the viewport. The doesn't know the size the image is going to be on the page, or anything else.

That works really well if the image is basically 100% of the viewport size, or nearly 100% of the viewport size. The thing to remember about these image sizes and sources is that we don't have to be precise. We've got a little wriggle room because the browser can resize things.

Say the viewport is 1024 pixels wide and the image in the page is 250 pixels. Knowing the width of the file sources and knowing the viewport size doesn't give the browser enough information to download the correct source. It's going to grab a really huge image when, in fact, all it needs is a small image. Because the image is small in that page.

Jen

Because it has not parsed the CSS yet, it doesn't know what the layout is yet.

Jason

Correct. Which is why every time you use srcset, and you're using the width descriptors, you also have to use the sizes attribute. sizes allows you to write another comma separated list to provide the browser with information about how big that image is in relation to the viewport at different viewport sizes. The way you do this is, you have a comma separated list, and the first in each value in that comma separated list is a media condition. A media condition is very similar to a media query but not exactly the same. A media condition doesn't allow you to do things like print and screen.

But it does allow you to say max-width. And you can have a value. That value can be whatever sort of media query that you normally do. Then a length. That length could be 100vw — 100 viewport width, which basically means it's always going to be 100% of the viewport. It could be 33vw — a third of the viewport. You could say explicitly it's 250 pixels because you know it's going to be that size and it's not going to change. You could even use CSS calc() to calculate it.

The goal is to provide the browser with information about how the size of the image in the page changes as the viewport size changes so that it can immediately — before it even knows the CSS or JavaScript — look at that list of sources, combine it with the information it knows that you don't know, like display density, hopefully down the road, whether it can increase the size of an image according to network speed or user preference and pick the right image.

That's sizes and srcset, just applied to an image, no <picture>, no anything.

Jen

So you're saying, "Hey srcset, here are my files, these are the widths of these files," and you're saying in the sizes attribute, "Hey, I've got four different breakpoints. At each of these breakpoints, this is the size that you're going to be filling up with one of those files."

Jason

Correct.

Jen

"You still get to use the math, browser. I'm figuring out which file goes into this space, but I'm going to define some spaces for you, right now, here in the HTML, because you haven't downloaded the CSS yet, you're not going to be able to get that information out of the layout. And you need to know that information right now, so here it is."

Jason

That is exactly what's going on. When I read articles about the responsive images specification, there are a lot of people who complain about the complexity of the solutions.

Jen

Yeah, this part is complex.

Jason

[00:33:44] Yeah, and I get it. In an ideal world, we would find something that wasn't quite that complex. But I also feel that images on the web are more complex than people give them credit for. In the early days of the web, Lynda Weinman wrote all of these books. I think she had five or six different books on the graphics. The first one was 250 pages, the second was nearly double the size and it was only a year later.

Putting images on the web has always been complex. Trying to find a solution for how to support responsive images but also have the browser be able to make decisions so it can download stuff quickly, is a really difficult problem.

I understand it's complex. We have a complex set of problems that we're trying to solve, and consequently we have some complexity in this. But gradients on the web are complex. Everything's complex the first time you use it. The tooling we have will get better so you won't have to do as much manually.

There is one bit of criticism I have sympathy for, but I think it's unavoidable. It is the fact that in sizes, we are providing some information about the presentation layer in markup. Which ideally...

Jen

You would not do.

Jason

Yes.

Jen

Because if you change your design, you've got to come in here and change this little bit of markup.

Jason

Correct. Basically, it's just, people have been working on this for four years now.

Jen

There's no getting around it.

Jason

Yeah. On the plus side, as I mentioned earlier, this does not require the same level of precision as your design media queries do. There's more of a fudge factor in what you do. If you say the image is a third of your screen width from 600 pixels to 1200 pixels, but in reality, in that continuum, it really does from 27% to 33% — it's stretching across that — you're going to be totally fine. There's nothing about this what will break. In your redesign, if that range doesn't change in those numbers because there's more fudge in that change, you're going to be good.

This is another indication that the tools that we use are going to have to start handling a lot of this for us. Adam Bradley recently said, "Save for the web is dead." [Jen laughs] I think he's spot on. The same way it doesn't make sense for individuals to create videos at all of the different videos and bitrates and codecs that are needed. It no longer makes sense for individuals to do the same things with images. Unfortunately, it's going to take us a little bit of time to get the tooling in place. But I do believe that's where that is headed

Jen

Back to the confusing part. [Jason laughs] And then alt! I'm looking at responsiveimages.org. I'm looking at this example, which is helping me. There's this simple markup here. I just have to toss in there that you also add an alt attribute to your image.

Jason

[00:40:08] The alt attribute should be on the <img> element and it will be on the <img> element regardless of whatever else you do. As a matter of fact, one of the early arguments we had when we were talking about the <picture> element was whether or not the alt attribute would need to change. The conclusion we came to was, if you have a situation where the alt attribute would change based on the image source, then it's not really a single image. It shouldn't be solved with <picture> or srcset or any of these things.

Jen

If the image itself is different enough: "Here's an image of a dog. We made it an image of a cat at this other size."

Jason

Exactly. You could do that with srcset and <picture>. Either one of those. You could switch from dog to cat to butterfly based on viewport size. That is an erroneous use of those solutions.

Jen

So don't do that.

Jason

Yes. Do not do that.

Jen

That could be something where you use JavaScript, perhaps. If you're doing something crazy and you want the content to change based on the behavior of the use.

Jason

Exactly.

Jen

That's interesting. I toss that out there just because I see people too often ignoring the alt attribute. It's interesting that was actually a big debate about it.

Jason

Well, I mean, big debate...

Jen

A small debate.

Jason

Yeah. I'm exaggerating. It was one of the things that came up in discussions. Especially when we start talking about art direction versus resolution switching. Which might be a good thing to talk about at this point.

Jen

Yeah.

Jason

[00:42:12] There are a bunch of use cases that the reponsive images community group documented. They break it down a lot further, but I tend to look at it as two use cases. [Both laugh] Sorry everybody.

I think of it as resolution switching versus art direction. Resolution switching it what you will face 90% of the time. All that means is, you have the same image, same aspect ratio, same everything. But you need to provide different sources of it based on the display density or the size that is used in the page as the viewport size changes. Those sorts of things. For whatever reason, it's the same image, it's just shrunk or larger. That is resolution switching. When you're doing resolution switching, you do no need the <picture> element. You need the <img> element with srcset and sizes. Or srcset on its own, if you're just doing display density.

Jen

I think the display density might be very common. If you've got a giant hero graphic across the top of the page, either edge-to-edge or within a column of content with a constraint with margins around it. That's the image that you are going to want to be gorgeous and huge and beautiful on a giant retina screen. It might be 12" of triple density imagery and that's a lot of pixels.

But a lot of images are not that kind of hero graphic. A lot of images are icons of users and pictures of people. You go to a conference website, what have you got? You've got all of these pictures of the people who are speaking at that conference. Those images are not blown up to be 12" wide. They go between a half an inch and 2" wide, by the time you use them in all sorts of different situations. You can load the 2" one and squish it down to a half inch space. It's retina screens and 3x screens... that's where I feel like I've started to want to use... to me, maybe that's a 60% use case. A 30% use case is, you need a big image that's going to swap out. You're using a differently sized image.

Jason

I think there is quite a bit of that use case. In our use, I would argue the vast majority of the time, if we're going to use srcset, we're going to use width descriptors. The display density descriptors — the x descriptors — are binary in a way that doesn't give the browser as much flexibility as it might otherwise have. It's like, I declared this to be a 2x image. But the screen is actually 1.8x or 3x. This is sort of weird: I would rather think about the complexity of the sizes attribute than I want to think about what display densities I want to support.

Jen

Do you trust the display density? Do you feel like a 1.8x Android device is not going to know that it should load the 2x image and round down?

Jason

I have to go look to verify this. Everybody should double check me on this particular statement. I believe if you have a 1.8x display and a 2x image, that the browser must download the 1x image and not the 2x image. I believe that it is a must, not a should, not a maybe. But that could be entirely wrong. They may have discretion there. One of the nice things about srcset with width descriptors is the browser has discretion. Maybe srcset with x descriptors works the same way. I need to double check.

You do have that same sort of challenge. I have a strong bias towards giving the browser information and letting it make smart decisions. I feel like by giving it the widths, I'm giving it all of the same information it might need with the x descriptors but I'm giving it more and I'm giving it the freedom to make smarter decisions on behalf of the user.

Jen

That's interesting. As you're describing it, I'm beginning to think, "Ok, 1x, 2x, 3x is going to be simpler for me to understand. I don't have to worry about the sizes attribute. i can focus on cutting three images, making three sizes, putting x different x descriptors on it."

It feels like using width descriptors means I'm wrangling more of the power and control. I'm saying, "I don't trust you to pick the right width descriptor. I'm going to tell you which width descriptors to use. It's harder for me because I have to do some of the work rather than you doing the work." I'm trusting you less, in a way.

But what you're describing is probably more accurate. We're trusting the browser more, we're just giving it more information to make its own decision. This also feels like one of those things that is nice in theory, but let's get three years of using this and then we'll all know which one we like better. [Both laugh]

Jason

There are use cases where the width descriptors are insufficient. If you end up in a situation where you've got an image where you can choose between x descriptors and width descriptors, you get to pick. There are going to be cases where it just won't work, and you've got to go the other way.

Jen

It seems like if your image doesn't change in real world sizes — if you took a ruler and held it up to a bunch of different devices, it would be about the same amount in inches or centimeters. This is a 5 cm image no matter where it shows up. Maybe it changes a bit from 5 cm, but it's not all of a sudden 20 cm. Then x descriptors might be totally fine. But you could also use width. If you get to a point where the image is growing or shrinking in the layout... frequently, it's bigger on a smaller device and smaller on a bigger device. That happens a lot in responsive web design. You're going to need to use width descriptors. The browser can't predict which one to use, because it doesn't know it's going to be bigger for smaller sizes and smaller for bigger sizes. You have to tell it that.

Jason

Spot on. The other thing that is kind of counterintuitive but can happen. Say you've got a responsive design where the image does change changes, but it changes sizes in an adaptive way. Where it's a fixed width. Maybe it's 300 pixels wide until it gets to 600 pixel viewport sizes, then it switches to 400.

Jen

It jumps.

Jason

Exactly. You've got these jumps. You know exactly what those jumps are. But you can end up in this strange spot. When you do the 3x version of that small image, you realize that it's actually the same as the 1x version, two breakpoints down. By doing width descriptors, you end up providing less markup and less sources.

Jen

There's some efficiency because you can reuse images for different sizes.

Jason

Anyway. We're sort of ratholing a little bit.

Jen

I know, I know. We're now 50 minutes in.

Jason

I'm sorry!

Jen

I'm drawing some of this out and getting you to repeat some of this and add clarity. At least for me, when I am trying to learn a new technology, it's easy to Google and look at a tutorial and read some code. What's hard is to really wrap my head around why I want to use which thing. That's frequently the thing that's not well-explained or well-documented. It's the hardest thing to teach, it's the hardest thing to write. It's very easy to write, "Do it like this!" So you're also teaching me because I don't know this stuff yet and I have been confused trying to implement it myself.

Jason

[00:51:59] I just want to say that this is exactly why the talks and workshops that I'm giving are all on this topic this year. They're all centered around, here are the problems and here is the logic for why you use things. I'm about to kick off a series of blog posts, basically responsive images 101. Where I go through the same stuff again. Looking at it from an implementor's perspective rather than a standard-setting perspective. This is the first time I've ever been involved in a standards process, so I still feel like I'm an interloper in that space. I'm very much in the space of, "How do we make decisions about what makes sense on this particular design?" So look for that stuff soon.

On <picture>, resolution switching is that first use case. The second use case is art direction. Which is basically, you've got an image and you need to change it when it gets to different sizes. Maybe you're cropping the image closer so you're able to see more of the image. If you go to shopify.com, they're using the <picture> element for exactly this use case on their homepage. They've got an image of one of their customers. She's standing up on wide screens because there's enough real estate to do that. In small screens, they've cropped her so it fits into the layout. [Laughs] I was going to say they chopped her in half, but they're not magicians.

Jen

Right, they crop across her thighs on the bigger screen and on a smaller screen they crop her at the waist.

Jason

Exactly. Another example of art direction is text. I've written recently about hero images, which oftentimes have text inside them. if you shrink down the images, the text becomes unreadable. Ideally, text wouldn't be part of the image, blah blah blah... but in reality, in the real world, this happens. There are people who have to create new promo images on a daily basis. Moving all of that into markup is just not practical. So how do you make decisions about that? That's art direction. When does the text become unreadable? When do we change it?

For art direction, that is when the <picture> element makes sense. The <picture> allows you to declare a series of different sources. You've got <picture> as a wrapper, much like the video element. Inside of it, you've got a series of <source> elements. The <source> elements can contain a media attribute. That media attribute is a full media query, with all of its power. Then srcset, where you declare the different sizes — you can use width or x descriptors — inside the <picture> element is also the image. We always have an <img> element. That is a requirement. That's how picture works. Those media attributes are declarations to the browser. You're not giving the browser any choice. The browser has to use whatever images you've declared at the different breakpoints.

That's the art direction use case of <picture>.

Jen

Is the <img> element inside the <picture> wrapper a fallback? When does that get used? Does that get used when none of the sources are used?

Jason

It's kind of both. If none of the sources match, the <img> element is used. But if one of the sources matches, the <img> element is still used. [Laughs] Going back to that analogy about the <img> element being a box into which the sources are used. Say the first source matched. This media query is correct right now. Then we look at srcset and we say, "Ok, now we're going to grab image A." The browser will take image A and it will set the current src of that <img> element to image A.

Jen

So it rewrites the HTML.

Jason

It doesn't rewrite the HTML.

Jen

It renders the HTML differently than you've written it? [Laughs]

Jason

The HTML remains the same. If you view source, it will always be the same. If you're in JavaScript and you take that src attribute and say, "I found the element, I get element by ID and I switch the source of an image element to a different image." The DOM will change. If you look at it in the inspector, you'll see that the properties have changed. But the actual HTML markup that the browser downloaded is still the same. It's one of those things.

The same way in which you've got classes. Just take any . You've got a with a bunch of classes on it. In JavaScript, you switch classes. You're not going to see that in the view source. But you would see that as you look at the element in the inspector. Does that make sense?

Jen

It does. It's a very geeky, technical clarification that the original, native HTML is untouched. For all practical purposes, what the browser renders and what you see when you inspect element with a web development tool is an altered reality of that HTML.

Jason

Yeah. From a JavaScript perspective, it would be img.currentSrc is the JavaScript value that you would look at. I don't know whether there's a place in the inspector. If you're right clicking on the image, that you can see that value. If you're looking at it from a JavaScript perspective, you watch that change.

I've got a demo page that is one of the many blog posts that I need to write. It's a demo of the <picture> element and you can resize the browser and watch that src change. It's sort of a stupid demo — it's incredibly simple — but it helped crystallize in my mind that <img> element is not quite a fallback. The <img> element is actually what is being viewed on the page. Even though you've got <picture> and srcset, the <img> element is the thing that is being displayed.

Jen

It's really interesting. I'm looking at this example of responsiveimages.org. In the markup, there's a <picture> element that wraps everything. There's two src elements, each with stuff going on. After those two src elements, inside the <picture> element, is this image. It says <img src="fallback.jpg">. In this example, there's a file called fallback.jpg. It sounds like if this HTML were run and the conditions were such that one of these sources were true. If we were in a mid-width of 40ems and we were in 1x, then big.jpg is the file that's listed in the srcset of the src. That big.jpg would replace the <img src="fallback.jpg"> in the rendered HTML. The rendered HTML, in whatever universe it lives in, it would say <img src="big.jpg">.

In a way, this whole package of <picture> is a whole bunch of data, but really it's going to continually change where the <img src=""> is pointing to. It's not going to eliminate or replace the <img src="">. It's going to modify it.

Jason

Yup. It is exactly that. We should probably change the name of that image. I might be a little bit further along in understanding what is in the new responsive images stuff, but I'm still learning things. I'm still thinking through how I'd use this stuff. For quite some time, I was thinking about it as a fallback. Other people were thinking about it as a fallback. Then somebody chimed in and said, "Actually, it's not a fallback. There are cases where you have a fallback. If this isn't used, then this other thing is used." This is always used. This is required. You have to have the <img> element. Everything else is being put into that <img> element.

Jen

It feels like the <img> element is not a fallback. That's not the fallback. The src attribute, where it says on the page src="", that's not a fallback. But the actual file that's specified — in this case, fallback.jpg — those characters in the code, the actual name of the file, is a fallback and will only be used if it's not replaced. But it will get replaced. The <img> element and src attribute will continue to be there. Normally, a fallback is completely ignored when it's not required. In this case, the fallback is the filename. The fallback is not the whole line of code.

Jason

[01:02:56] Yes, absolutely. There's one other use of <picture>. Do you want to talk about it briefly?

Jen

Sure.

Jason

Ok. <picture> for art direction. You've got the media attributes in it. There's this other thing that got added to <picture> late in the process. Which I think is one of the most brilliant pieces of the new responsive images stuff. That's the type attribute.

On any given src, you can declare the type of image with a MIME type. You can declare it as a WebP, SVG, PNG, or whatever it is. Much like video, where you declare different codecs, if you use the type attribute and declare different images, the browser will ignore the types it doesn't understand.

Jen

Right. You could use WebP — which is an amazing, awesome, tightly compressed image format that is not widely supported — and it would be used on the browsers that support it and ignored by the browsers that don't. Those browsers could have JPG or whatever else.

Jason

Ding ding ding. Exactly. Sara Soueidan has written about using the <picture> element with SVG. You could say, your first one is SVG — because not all browsers understand SVG — but the browsers that do understand it will use it, and you've got a PNG fallback. It's all taken care of, <picture> handles it.

Jen

That will allow more innovation in image file formats. It's really hard to innovate image file formats. We've been stuck with JPG and GIF forever. PNG came later. Even using PNG24 was really hard for a long time. Nobody wants to use files that can't be used, but nobody wants to implement formats that nobody's using. So you get this situation where everybody keeps using JPGs. But now if you've got a technique for using them, even if they're not fully supported, then people will use them, which means browser makers will want them. Which means we'll have new images, beyond WebP.

Jason

To me, this is an incredibly exciting development. It's the first time where we might see competition amongst image formats.

There was a really great article, I'll send you the link. I will admit, most of the conversations around images is a bit dense. It was looking different image formats, in particular JPEG 2000 and the way that it handles alpha transparency.

There are two things that I didn't know about JPEG 2000. The first, it handles alpha transparency amazingly well. These examples are alpha transparencies where the JPEG 2000 file size is 24k. The same image in WebP is 56K. The PNG version — which is what we would be using if we wanted to pull off that effect without either of those formats — is 300k. Huge difference.

Second thing I didn't know about JPEG 2000 is that it's supported in Safari.

Jen

To me, you say JPEG 2000, I think, "Do not do. Do not. Do not. Turn around. Go the opposite direction. Do not use."

Jason

Exactly. I've been in the same place. It's interesting, it's kind of cool, but I can't use it anywhere. Now I'm like, I could. I could use it with the <picture> element. If I needed to do really complex alpha transparency now, I might do a JPEG 2000 version. For browsers that support JPEG 2000, that's going to be lightening fast. Then I'd do WebP. Then I'd do PNG.

Jen

Because WebP is Chrome and Opera. JPEG 2000 is supported in Safari. Then you're just left with IE and, increasingly, yet again, who's behind? Firefox? Hello, Mozilla. You've got to catch up, Mozilla. IE is going to crush Firefox this year, I feel like. [Jason laughs]

Jason

[01:08:12] One point to make about these file types. You can use the <picture> element to do image type switching without having to do the media attribute. Doing so, unless you're doing art direction, you should avoid the media attribute. Because the media attribute is the thing that switches from the browser making smart decisions to you declaring to the browser, "You must use this image." You could have a <picture> element with multiple sources and the type attribute and then srcset, or srcset and sizes, or whatever makes sense. No media attribute. The browser would pick the best type based on the first one that matches. Then it would go through and run all of its srcset and sizes rules and pick the correct image of that variant and put it in the page.

Jen

In a way, you're contradicting what we were saying before. We were saying, use <img srcset> unless you're art directing. If you're art directing, use <picture>. Now we're saying, use <img srcset> if you're not art directing and you're only going to use one format of images. You can use <picture> for art direction or you can use <picture> for using multiple different image file types, if you want to use WebP and JPEG 2000. Or if you're combining them, of course.

Jason

Who knows what will happen with this new ability to do image types. As you said earlier, we need a couple of years of people experimenting with this stuff to really know. Based on what we understand of the current use cases and usage we see in the wild, 80-90% of the usage is still going to be a single image file format and the resolution switching use case. Which means you just need an <img> element, srcset, sizes, or just the <img> element and srcset. The media attribute is really the thing that makes the switch from art direction versus resolution switching.

What I've found is, until people become more familiar with the different capabilities of responsive images, that it can be confusing. "You can use <picture> for non art direction, but if you're using filetypes..." I wrote a blog post: Don't use <picture> (most of the time). There were a couple of reasons why I wrote it that way.

One, the specification is called the picture specification. Even though, inside the specification, it defines srcset, sizes, type, and all of these other things. The vast majority of the time, you don't need to use <picture>, or more realistically, you don't need to use the media attribute. Doing so was actually harmful. But everybody heard picture for so long that I feel like we've got a little bit of re-education. Until this point in this podcast, I've intentionally never used picture syntax or picture specification. I've always said responsive images syntax or responsive images specification. I'm trying to train myself not to say it that way. We see this right now. We see most implementations of people experimenting with responsive images are using picture. Oftentimes, they're using <picture> with media attributes and declaring things, when in fact, really what they're solving for is resolution switching, and they should be using srcset and sizes.

Jen

Right.

Jason

I fully acknowledge there's a contradiction [laughs] between what I said earlier, "You don't want to use <picture> unless you're using it for art direction." In reality, it's the media attribute that causes it. You have to get to that point where you understand the difference between the two and why you're doing one or the other until you can muddy the water a little.

Jen

I think in the evolution of this conversation over the last four years, for a long while, there was a big debate. From the outside, a very simplistic version of it was the sense that web designers and developers banded together, "Yeah! Rah rah! We need a new element. We need superpowers. We need to do something awesome," and there was resistance from the browser maker side. A lot of saying, "No, let's keep what we already have. Let's keep it really simple. We just need to make this one little tweak." The rah-rah web developers wanted <picture> and the, "That's too hard, we don't want to do that much work" rumor — which may or may not have been accurate — was advocating for <img srcset> (not <picture>).

All of that got worked out. It felt like we ended up with everybody getting what they needed. Something that could be built into browsers, something that would not be slow, something that would let browsers render and focus on performance and give developers and designers the flexibility and different use cases that we need. It feels like everybody involved worked really hard and came out with something that is as good and close to awesome as we could possibly get.

Now here we are, after all of that. The word "picture" was evangelized as part of an effort to say, "We do need this level of complexity. We do need these use cases like art direction." Now everyone's like, "Yes, picture! Use picture!" [Jason laughs] Then you see blog posts that are advocating for srcset, and it's like, "No, srcset is evil! We hate srcset!" [Both laugh]

The Drupal module [for example] is called the Picture module. I tried to use it on The Web Ahead website to do <img srcset>. I just could not. I gave up. I was like, "That's it. I'm out of here. I'm done." [Both laugh] It may be that you can and I couldn't figure it out. It may be that you sort of can but it's badly documented. It may be that you can't and the picture use case is really only what's possible.

Jason

I think it got added to got module very recently. Depending on when you were doing it, it's a really good chance that it wasn't there.

Jen

Maybe. It seems like the last release was December 30, and then one on February 20. I don't know. I don't know if it was before or after December 30. It was right around that time.

Jason

I really do think it's this year. That February 20 release may have been the one that added srcset and sizes.

Jen

Then that would be why. It was definitely before February 20. In the end, I was like, "F everybody! I'm out of here!" I went back to making everything be a little too large but as small as I could. Shrinking everything down around 80%, sending things that are computed for 2x for retina, and just sending that to everybody. Trying not to use images very much on the website at all. That's been our strategy. I think that's been everybody's strategy for the last several years. Don't use images for things like drop shadows and cutesy background papers. That's why everything is so clean and simple. Use as few images as possible. Make them as small as you can, but think about retina screens. Make them big enough and live with one set of images. But now is the time, like you said, to learn this and really start reconsidering that and using these tools.

Jason

The Drupal stuff is a good example. They didn't yet have the stuff that you needed. There's complexity now but I don't think that two or three years from now we're all going to be facing this complexity every time we do stuff. We will get better at understanding the syntax. Every piece of what we do is complex in some way. The first time we're introduced to new CSS, it takes us awhile to understand. I don't think the markup and pieces for responsive images are anymore complex than flexbox or gradients or things like that. There is inherent complexity in images. That's the piece that's a little bit harder to wrap your mind around. Which use case am I solving for? Where do I put the image breakpoints?

The responsive images community group has an official plugin for WordPress. I've got this spreadsheet of image resizing services. Many of which will also attempt to do things like handle markup. I look at those things and think, in the long run, we'll think about these things at the moment that we're designing templates. But they'll be codified into the tools that we use. Maybe they're in Grunt or Gulp or our CMS. In the long run, it seems unlikely that we're going to end up doing a lot of this stuff manually. Like video, images are moving towards the space where it makes more sense to let the computers do the smart stuff for us.

Jen

That is the beauty of something like Drupal. Every time I ever do a Drupal site, my intention is to mandate that everyone upload the largest image possible. Give me something that's 2000 or 4000 pixels wide. Lets just get the system to ingest really big originals. Then the robots with ImageMagick running on the server chop everything down to the smallest thing possible. Compress everything. That's without even getting into responsive images. You've got an image or your blog post? Over here you want a square teaser thumbnail, over here you want a rectangle thumbnail, over here you want a big one, over here you want a small one. You get the computers to cut all of those different sizes. That was true with fixed width websites before responsive web design or mobile. If you've ever tried to teach editors how to prepare content for the web, getting people to open Photoshop and save for web, that is hard for people. They fail to do it successfully. They don't understand what a JPG is or when they should use a GIF or when they should use a PNG. Teaching people to do that manually is really laborious and very prone to errors. Automating all of that is much better.

Jason

Exactly.

Jen

[01:23:33] I wanted to ask, <picture>, <source>, and <img srcset>. Some of these things are really similar. You can put a srcset attribute on the source element inside the <picture> element, or you can use a srcset attribute on the <img> element directly. Are those two things identical? Do they work in exactly the same way?

Jason

They do work in exactly the same way. When you have a <source> element, you cannot have a src on the <source> element.

Jen

You only have a srcset.

Jason

Correct.

Jen

For the <img srcset>, you have the sizes attribute. Yet on the <picture> <source>, you have the media attribute.

Jason

On the <source> element, you have a series of optional attributes. You've got the media attribute, type attribute, srcset attribute, and sizes attribute.

Jen

So you could have sizes on both. I assume sizes works identically on each.

Jason

Correct.

Jen

Or else we'd really go crazy.

Jason

Whenever you see srcset, it's going to do exactly what you expect it to do. Whenever you see sizes, it does the same thing as you expect it to. There's no changes in behavior. When it's on a <source> element and there's also a media or type attribute, the browser will evaluate those two before it pays any attention to srcset or sizes.

Jen

It's a conditional?

Jason

Yeah. Media conditions are, does the browser meet this condition? Is it true or not true? It starts going through this list of sources and finds the first source that matches the media condition. And is a type that it understands. Then it will look to see what srcset and sizes say. But until that point, it ignores them. If they are set. If type or media is set. It looks at those first to make a decision about whether or not this <source> element is one that even matters to it.

Jen

So type and media would only be inside <picture>? You wouldn't have type or media on an <img srcset> directly?

Jason

Correct. You would never apply them to the <img srcset> element.

Jen

In a way, what you're getting with <picture>... with <img srcset>, you have one stab to do one set of srcset and one set of sizes. With <picture>, you're getting multiple stabs. You can do your <source> to do srcset and sizes of one group, and another <source> to do another set of srcset and sizes to do something different. In order to decide which <source> to use, you've got the conditionals of media and type to specify, "Hey, this is the <source> that you want to use. Ignore those." Does it pick the first or last one, like CSS?

Jason

It picks the first one. Just like video does.

Jen

And then ignores everything after it?

Jason

Correct.

Jen

Interesting.

Jason

God, I hope I'm right about that. [Both laugh] I have this pang of, "Oh my gosh, am I crazy?"

Jen

Here we go. If anyone has a correction, because you know anything we've said is wrong, you can run — run, immediately — to thewebahead.net/99 and put a comment on the website.

Jason

I am 99% certain that it's the first one that matches. It's got to be the first one that matches. But having the question put to me that way... [Both laugh]

Jen

It's a quiz! It's a pop quiz.

Jason

All of a sudden I'm like, "Wait. Oh no. I'm wrong!"

Jen

Yeah.

Jason

[01:28:02] One of the problems we have when we look at this stuff is, we wind up looking at that last bit of complexity. We've got an example where we're supporting different image types. We've got media attributes in there because we're doing art direction. We've got different sources in that. We've got srcset and sizes and this huge bit of markup. That's just not going to be what you commonly use.

I created this crazy flowchart where it's like, "If you're doing this, ask yourself these questions." It was spaghetti by the time you go through all of these things. I realized that wasn't the right way to learn this. The best way to learn it was to start with the image. Because it's always required. It's the core. Then add one srcset. understand what srcset was for. Then add on sizes and understand why sizes was necessary because of the pre-parser. Now look at art direction and understand how src attributes and the <picture> element contribute. Then get a sense of what type does.

For any given image, you may not need any of that complexity. You may need just a small piece of it. You could find yourself in a situation where you need all of that power. These are just tools. It used to be that you'd go out to your workshop garage — your metaphorical workshop — and you'd look at the tools for responsive images. Basically, the only thing you had was a screwdriver. It was a flathead. Good luck if you had a nail or a Phillips head. Now they've handed you this tool chest with several different tools that you can use together or just one or whatever makes sense.

A couple of weeks ago, I wrote about responsive image audits. Which is something we've been doing for clients as we help them move into responsive designs. We'll go across their site and look at the different templates. We're always looking at a template level and the types of images that are used in them. We'll ask ourselves a bunch of questions about these images. How are these images produced? What is the variation between the smallest size and largest sizes? Do we need to support different image types? Is this art direction or resolution switching? I documented the questions that we ask. I included an example of a spreadsheet that we came up with for one of our clients that was six rows of different types of images. Categories of images that were used throughout the site and how we would solve them. What the solutions were. What syntax we'd do based on what we had learned about them. There was one category of images that was 90% of the images on the site. We only had a small subset of images that we had to worry about art direction on, that we needed <picture> and media attributes.

Once you understand how the different pieces of the syntax work. Going back to this metaphor, you have to understand what the tools are for. But the reality is, when you're working on projects, there's a handful of tools that you use all the time. You've got the other tools that are only for special uses. I think that's where we'll end up with a lot of the responsive images stuff. We won't be always grabbing into that toolbox to grab five items at a time and combine them all together. We're more likely going to go in and grab that one specific screwdriver for the type of image that we're working with.

Jen

It does feel like the kind of thing that's much easier to do if you have a specific task at hand. I am building this website, I need to support this set of images, I'd like them to look good on retina screens. What do I want to do about that? Tackle a little bit at a time until we get the hang of things.

Jason

Exactly.

Jen

Thank you for coming on the show.

Jason

You're welcome. Thank you for having me.

Jen

I've already gathered a bunch of links and asked you to make sure that we get the rest of the things that you were talking about today. People can definitely follow up and find more information about this stuff. And I do mean it. If you have a correction, add a comment to the show. If you have an example or tutorial or some kind of resource or something that you think is pretty awesome. Maybe something you made or found that helped you learn. Definitely put those kinds of things into the comments for the show. People can find them the same way they find the show notes. All of that stuff, along with the transcript and everything else, is at thewebahead.net/99.

I should say thank you to our sponsors today, Code School and Squarespace, for making the show possible. This is 99. Next week will be episode 100. It will be good. It's a little depressing, the end of it, but in a really good way. Definitely subscribe. If you haven't subscribed to the show, subscribe to the show through anything you want. RSS feed reader, podcast application. The Web Ahead will work on any of them. As always, thank you for listening.