Transcript

Jen

This is The Web Ahead, a weekly conversation about changing technologies and the future of the web. I'm your host Jen Simmons, and this is episode number 48. I first want to say thank you so much to the sponsors of this week's show, An Event Apart and the Google Developers Live Chrome Dev Channel. We'll talk more about those later in the show. So this week is going to be another show in the Web Behind series co-hosted by the fabulous Eric Meyer. Hi Eric!

Eric

Oh, Hi Jen! Fabulous is not a word most people use to describe me.

Jen

It threw you off there?

Eric

It did a little.

Jen

Surprised you?

Eric

I mean, once they've seen my fashion sense then fabulous is not a word that comes up ever again. But I like it.

Jen

Well, CSS guru.

Eric

Alright, lets just get off that because I'd actually like to welcome this week's guest, Danny Sullivan is the founding editor of Search Engine Land and Marketing Land, he also writes the Common Sense tech column for cnet, and is in charge of the Search Marketing Expo conference series, he's been doing this whole search watch thing for many many years now, and I'm really excited to have him on today's show to talk about the dawn of web search, so hi Danny!

Danny

Hey, thanks for having me

Eric

Yeah, we're really glad to have you. You're located out there in California, I understand.

Danny

That's right.

Eric

Yeah, we've gotten suddenly very cold.

Danny

I hope you'll warm up. It's cold here too. It hits us first and it moves off that way, so what I can tell you is we did have some warm weather, so hopefully it'll make it out your way soon.

Eric

[00:01:46] Yeah, I think probably by April we should get some of that. So as I said, hopefully we're going to explore the beginning of search because, as we were talking before the show, as Jen was pointing out, probably a lot of people listening to this podcast, or even a lot of people on the web who are not listening to this podcast, don't remember a time before Google, let alone a time before search engines on the web, and I think that's actually an underrepresented story. Most people don't really think about it. Where we usually start is by asking people what you were doing when you first encountered the web and what you thought when you first saw it.

Danny

Sure. I first encountered the web reading about through Wired Magazine and this was probably around 1994 and there were these references to the web. It sounded intriguing, but I didn't really know what it was. I went to a conference around the end of the year, and had seen it, and just went, "wow." I thought it was incredible. It was going to be this huge new publishing opportunity. It was just so obvious that it was going to explode, and I was immediately feeling frustrated that I was at a newspaper which, like most of these newspapers at the time, didn't know what to do. I can remember being in a meeting where we were discussing our online options and what we were going to do to be online and the debate was going between, "do we do CompuServe, or do we do Prodigy, or do we do AOL." Kids, you can look up CompuServe and Prodigy later, "or do we do MSNSearch," and I thought, "I don't know, I'm outta here. I've got to do something else because they're not going to get this." And so I joined up with a friend of mine and we started putting up people's websites online, and there's twists and turns to the story down the way, but it ended up with me being involved with the writing that I do on the internet now. It really was one of those things where you just see something and you're going, "This really is going to change the world," and it really did.

Eric

Do you remember, was anything specific that made you think that? That you saw.

Danny

It was clear that easy for anybody to publish and that there were going to be easy ways for people to interact with business and companies. I can remember things like wanting to go to a store and wishing I could just email the store to find out from the store what their inventory was, and oh my goodness, here was this service that lets you get all that information and potentially put it up online. I remember trying to start a magazine with some friends of mine we wanted to do a magazine for people who are Gen X, and it was just so complicated if we were going to do it we were going to have to print it out, you're going to have to get all the production that was involved with it, you know, how do you get it mailed out, and that sort of way. And the web was a printing press for anybody. There wasn't going to be that kind of hassle with it. It was so easy for people to put information up online that it was clear that it was going to explode, and that's when it really stuck in my mind.

Eric

Yeah. I think that a lot of people had that feeling. I actually remember the last Web Behind guest we had, Jen Robins was working at O'Rielly and I think had a somewhat similar feeling. As they did at O’Rielly at the time. That ease of publication was which is [ 5:27 inaudible] because sometimes when we look back, there was no tumblr, there wasn’t blogger, and there weren’t nearly the number of firms there are now that are, “we’ll put you on the web for $49 or whatever. You and your friend formed effectively, what, a web design firm? Or what did you call yourselves?

Danny

When we got started originally?

Eric

Yeah.

Danny

It was web design and production. The company was called Maximized Online. My friend’s company was Maximized Software, so we started an online division. We called ourselves at the time, Internet Presence Provider, IPP. [laughs].

Eric

Nice.

Danny

We went into it thinking first of all that big ad agencies put all the big companies online so we put together packages that were aimed at local businesses. I think it was $150 and we would build you five web pages, which would still be a bargain today. We build you your website, we were hosing you for a month or a year. I can't remember what it was, but we were hosting you. We were concerned that people couldn't be found if we were putting you online. We thought small businesses would immediately be asking, "Well, how is anybody going to find me?" so of course we were going to submit you to major search engines and directories at the time. And then we also created our own directory just for the local area we were at in Orange Country, California so that we could further help people understand that, "No, you're going to be found, we're going to get this stuff out there." And so we did all those sorts of things and our only mistake really was, aside from having not really done this sort of business before, assuming that local businesses would want this. And it turned out that local businesses thought $150 was a huge expense that they were not going to shell out. Instead they wanted to continue to spend whatever they did with their yellow pages and so on. And then alternatively, we had these big companies that started coming to us because they needed help and they'd heard about us. And we didn't really have anything in position to do all that kind of client management. It really wasn't the strategy we had started with. And by the time we had got ourselves situated to do that we really had run out of money to keep doing the online service any longer. My friend decided to focus on software and I went off to be a consultant and started looking at search engine stuff.

Eric

[00:08:10] So was that as a result of all of the work that you had to do to try to get business noticed in the first place? Like was that the progression?

Danny

To get me into the search engine things?

Eric

Right.

Danny

Actually, it was more of trying just to further improve our own processes. We were doing these sort of submissions, and in particular, it's one of those things you never forget, we had this one client who just felt like he should be ranked number one for Orange County. He had a small jobs board that he was starting everybody at the time, by the way, thought that they could just create an entire new businesses that was already replicated in other things. So for example, even though there were places and companies that would get you jobs already, you thought, "Well, I'll create the job service and I'll start if off this way in Orange County, and it's going to grow and take over the world." And multiple people had no experience and no business plans, no concept whatsoever of what they were doing. But in his case, he was frustrated because he thought he should be number one when someone searched for Orange County on whatever it was at the time, Lycos, a web crawler, and so on. And we did the submission to search engines, and we put in things like meta keyword tags and did the various little bits of information that were out there at the time, but we didn't have any good answers for him. Nobody really understood how these sorts of things works. So I spent a lot of time over the course of about two months looking at what the factors were that seemed to influence why somebody ranked better, what kind of changes made an impact. All the information that I found I ended up publishing in something I called, "A Webmaster's Guide to Search Engines," because I was trying to help....It was interesting to me. I learned a lot about the internet from other people, so I kind of felt like this was a way for me to contribute back. And it really was designed to help people understand that, "Hey look, there are these search engines out there and they don't operate the way that you think, they don't necessarily constantly come back to your website and they really care about your html title tag and those images you built on frames are an incredibly bad idea if you're trying to get found because they don't understand what that's all about."

Eric

Mmm, yeah. [00:10:20 inaudible] 'cause search engines weren't nearly as sophisticated as they are now.

Danny

Right.

Eric

[00:10:26] There was something else that you mentioned that I think maybe some of our listeners might have cocked their heads at a little bit. They would understand the word but maybe not the context which is: submitting to directories and building your own directory. That was really big, early on, right? Yahoo started out itself as an enormous directory, but it was a lot of people build small directories and local directories, and whatever happened to that?

Danny

Well at the time we had the little directories that were out there that, well not all of them were little, the big one, and in fact a lot of people forget, was the huge giant player was Yahoo. People these days talk about Google in the way that they used to talk about Yahoo, in terms of it being dominant, in terms of it being the gatekeeper of the web, in terms of issues and what they should or shouldn't list. And Yahoo was a directory. You would submit your 25 words, and you would submit the category that you wanted to be in, and you would hope that an editor would decide to include you, and there was not guaranteed time when it would happen, there was not guaranteed support, and if you did get accepted and editor might make a few small changes that dropped out important terms about your company, and then you weren't being found at all. But what happened was, it's sorta like saying, technically why they went away, a directory is like going to a library and trying to find a book by flipping through the card catalogs. For those who remember card catalogs, the book would be listed, you'd have the title of the person who wrote it, the author, and maybe 25 word description of what the book was about. So if you were going to the library and you were trying to find a particular book, you could search through just the summaries of these books, and that's what a directory did: it summarized what a book was about in say 25 words, maybe 100 words, so on. In contrast, the crawler based search engines, of which Google was one, there were other ones that were out there that predated Google, but the crawler based search engines, they were services that read every page on every book in the library so that when you did a search, they didn't come back and say, "Well, I found 5 books that maybe match your topic." They could come back and say, "I found five hundred thousand books that are all related to your topic." They were much more comprehensive and they were exceptional at finding the sort of needle in the haystack when you did that obscure kind of search that you were looking for. But the difficulty that they had is that they could also be overwhelming. One of the metaphors I always make is: people say that you want to have all the content that's out there on the web, and at one point search engines were fighting over who had they most webpages because you can't find the needle in the haystack if you only have half a haystack, that was the argument that was being made. And I would say, "Well, you can't find the needle in the haystack if I dump all the hay on your head as well." You get overwhelmed. I know there's the needle in there, and I know you've got the whole haystack, but I have to have a system for sorting through it. I'd have to a magnet or something. And that was the key thing that Google did. It came in with a different ranking system, it came in with a different algorithm, a different recipe for sorting through all those pages so that you didn't just get, "Here are 500,000 pages that match the search you just looked for in the library," it also said, "and here are the top ten best ones." And they really were good places to start. It gave you both the combination of the high relevancy you got with a directory by getting just a few matches and good quality matches, with the comprehensiveness of being able to search through the entire haystack.

Eric

Yeah. Directories, they always reminded me of the yellow pages, right? Because if you know what category and you know how to find that category, that can also be the interesting thing.

Danny

Right.

Eric

Then you can find things that are at least in theory directly relevant. Because I am not the handiest person in the world, so "My light doesn't work." If I know to look under "electricians", that's great, but if I look under "light doesn't work" in the yellow pages, then I'm not going to get it. I might find "lighting," but that's people who will sell me new nights, not people who will fix the light that I already have. Of course, I think part of the problem with directories, which seems obvious in retrospect, is that it wasn't going to scale if the web passed a certain threshold, which it very quickly did.

Danny

Right.

Eric

Yahoo would probably have a category for...I'm trying to think of something very common, just a category that there are going to be a lot of players in. Web design firms, there you go, the category of web design firms. That would end up being a very, very long list.

Danny

Yeah.

Eric

So I'm trying to remember. I have to assume that Yahoo tried to deal with that. Do you remember how they dealt with the problem. Did they break it down by geography, or...?

Danny

Well, they started adding on more and more categories, they did break things down by geography, but really the challenge that they were facing was simply not that they didn't have enough categories, but that there was so much growth going on. There were just so many sites that were out there. And so their immediate issue was, "Well how do you deal with the challenges where people say, 'you're not getting everything'?" and the answer was, "We're not trying to get to everything, we're trying to get to the best stuff," but even then they weren't getting to the best stuff because they were just so overwhelmed. Eventually they rolled out a system where people could do paid-priority submit, which meant that you could pay and at least get an answer in a week. They weren't going to guarantee that they would list you, but they would at least put it out there to say you could perhaps get located that way. That worked a little bit, but really before they could grapple more with the scalability problem they got overtaken by the crawler based search engines. Eventually, they ended up trying to buy their own crawling technology and basically did what Google was doing.

Eric

[00:20:34] So let's go back: Google came along and really changed the game, but who was before Google? What were the first search engines? Because to some degree I'd like to simply remember them because a couple of weeks ago I made a joking reference to Dog Pile

Danny

Right.

Eric

Saying that nobody listening to the show probably remembers Dog Pile. I don't know why, I think it was the name and the goofy little mascot, I remember Dog Pile.

Danny

We had a variety of ones that were out there. You had people like Opentext, which was a big one that had gone out there and crawled the web and found content that was out there. Infoseek was another one that eventually became the Go website by Disney. You had Lycos, which was one of the original crawling based search engines, named after a type of a spider, because the search used to crawl the web were all referred to as spiders, so they had these kinds of names sometimes that came with them. You had Webcrawler, that of course was crawling the web. Excite came along, and it also did some crawling. Hotbot, which was powered by a company called Ink2me that was creating technologies so that other people could go out there and pick out pages from all across the web. Alta Vista, which was at the time one of the new generation that quickly became very popular among people because it was even more comprehensive than some of those other crawlers that I named out there. It really brought up the number of pages that it was collecting from all over the place. And you had some of the directory. You had people like Magellan, which was like a Yahoo, where you would submit and you would be listed based on how people described your site. Snap was a c-net site that was out there for a while. Direct Hit. And then of course Ask Jeeves from way back in 1998, which was combining both the crawling of the web and also human editors that were answering a lot of the questions which is why it still has that sort of Ask and the long query invitation that you see today. Look Smart was another one that was out there. And a lot of these all predated or were contemporary when Google came along around 1998. And the interesting thing is when you look today, and people look at the landscape, you have people who think, "Well, it's always been Google, and Google created all this web search, and Google innovated all this stuff," and actually a lot of what Google did when it first came out was copying what all the other search engines were already doing, especially some of the features that they had. Even the concept that Google used to improve it's relevancy, which was to examine how people link across the web and consider links to be votes, that was being used by some of the other search engines as well, it's just Google had a better recipe of it. What's amazing to me is that, you look at things today, you have Google out there, which was really sort of your second, almost your third mover, third generation search engine, behind that you have Bing, which even though Microsoft Search itself goes all the way back to 1998, it really is sort of like one of your second youngest search engines out there because they redid the whole effort to build what was eventually called Live search, and they were going to have their own search crawling technology and it all morphed into what's Bing today, but it's still relatively new technology that they've used and put together, and effort. And that's it. Kind of like the two big surviving search engines out there were the new comers, even though we don't perhaps think of them as new comers now.

Eric

Yeah. So all those search engines you rattled off, that was what, about 1994 through 1998?

Danny

Most of them were from ninety...yeah some of them started to go back as far as '94, somebody like Lycos and Yahoo. Really '94-'98. And by the way, the one that you mentioned, Dog Pile, it was what's called a meta search engine, which was we at one point had so many different search engines that it became popular to have search engines that searched search engines [both laugh]. And they were called meta search engines. So you would do your search and then they would come back and they would give you answers from across all the different search engines.

Eric

Yeah, so if I remember, I hope I'm remembering correctly, it would actually show you where the result came from.

Danny

Exactly.

Eric

So it would say "Lycos has these hits, and Ask Jeeves has these hits, and..." Not that they were probably called hits, but it's the same idea. That's such a fascinating contrast between then and now. There were services that let you search multiple search engines. Probably because it works for money, but also in the hopes of finding your needle. Your needle's in a haystack, but you have 20 haystacks. So this service will look through all the haystacks in an attempt to help you find the needle.

Danny

That was their attempt. And that also goes back to the give people a little more of the technical mixed in with some of the history. When the first search engines came out, they were collecting millions of pages. Like, it was big if you collected two million web pages. When Alta Vista came along, it had an index of like 20 million, so it picked up like twenty or ten times the amount of pages that the other search engines had, and therefore it was considered to be much more comprehensive. And for the first five years or so of our search history, there was a real competition where everybody wanted to have the most pages. And in part you could get these differences because of that. You could have one search engine had an answer that this other search engine didn't because it wasn't big enough. Sometimes differences would happen because one search engine had a recipe that was better at searching through the web than another search engine had. And then over time what happened was they all just got so big that it really didn't make that much of a difference. And also, we just had more and more of the players starting to drop out.

Eric

[00:26:57] So what caused the dropping out typically?

Danny

Well, you know Google started picking up a lot of adoption because the results were so much better, and they'd actually put the investment into having money to be a search engine. When the search engines started, one of the challenges they faced was, it's funny to say now, but one of the challenges they faced was making money. People would come, they'd do a search, they'd click on a link, and they'd leave. That's not a good business model. Ideally, you'd like people to either pay you to be listed or people to click on things that are going to make you money. But people weren't paying to be listed in the search engines. And when Open Text started running paid sponsored listings, similar to what Google does today, people went nuts. They felt like, "No, you can't sell a listing, that's wrong. You can't do that sort of thing. It's a violation of editorial integrity and yada yada yada." So they backed away from it and none of the other search engines wanted to go down that road. This was like, 1996. And so instead, search turned into this kind of loss leader. All the major search engines started thinking, "Well, I guess we'd better become a portal." And the idea behind a whole portal was that, "Yeah you can do some search, but we're going to give you some free email, and we're going to give you a way to have, oh I don't know, maybe your stock portfolio, and we're going to get all the eyeballs coming back to our site each day." Sounds like the Facebook, model, right? "And we'll show them a bunch of ads, and the ads will be sticky and that's what people will do, and we'll be this sort of portal entryway into the internet. People will start everything out with us." So that sounded like a way that they should go, but they didn't because they weren't making money of search, they weren't really paying that much attention to it. And Google came along and said, "Well we think search is important, and we're going to have really good search results and we'll figure out the payment model down the line." And they started getting the audience and they were helped that actually a company called Goto which later became Overture, which later got purchased by Yahoo, went back into it and said, "You know what, we think you can sell search listing, and we're going to be a completely paid search engine, and we're going to build out a model, we're going to go out there and let people pay to be listed number one," or whatever, and it kind of reopened the idea that yes, you could sell paid listings. So, Google was able to kind of adopt that and start up when the environment was more friendly to the idea of carrying ads. If it had started in 1996, there's a chance it might not have existed just like the rest of them, because it might have gone down the different path and found it too difficult to shift around when they were already along that way. The other challenge that happened was most of the other players, if you had your own search technology, it was expensive to maintain. And so then people started looking about, could they outsource it. And they would try to get a company like Ink2me to pick it up, but those companies had like half a solution. Ink2me could give you the search listings, but they didn't have a paid solution to go with it. Goto could give you the paid solution, but they didn't have the listings to go with it. And then Google would come along and say, "Hey! Yahoo, how about you take our listings and maybe you could even take some of our ads, if you want," or that's what they did with AOL, well it's a big deal with them and netscape as well. They said, "Look we'll give you the whole solution. We're going to give you search listings, you don't even have to necessarily have to pay for them, you're just going to make money off of it, and we'll keep a little bit of the change off to the side." So they just picked up a lot of momentum, and it was just very difficult for the remaing people out there to compete at that point.

Eric

Yeah, if anyone wants to see a still extant of search engines that is hanging on to the portal business, I think excite.com is probably the best example I can think of, because it's still "Here's the weather, and have some email, and stocks right now, and here's Miley Cyrus video, and you can also search the web." And actually an extra bonus for those, the Excite homepage to me really feels like it's still stuck 10-15 years ago, because if you actually look at the source, it's all still built out with tables.

Danny

Gah! Yeah, that's it's very much a maintenance mode, we're not going to be messing with anything, we're just going to let it go as it goes.

Eric

But that's where a lot of them went, right? Excite was one of the ones you mentioned, and I immediately thought, "They're still doing it," still trying to be a portal to the web that happens to do some search. Or a portal not just to the web, but all the other...

Danny

Although Yahoo still is very much, I think, the portal model.

Eric

That's true:

Danny

It's sort of the portal survivor. And the challenge I think it's facing these days is that when I go back to Facebook, that has sort of turned into what people's place to start the day is. You can get the web, and get your information. So that's one thing that I think they're struggling with.

Eric

[00:31:59] Yeah, yeah definitely. All I can do is agree there. So there was a phenomenon that I remember from that '94-'98 period, which was that there would be these mass migrations from one search engine to another, like everybody would be using Lycos for a while, and suddenly word would spread out like, "Oh my gosh. Alta Vista is so much better," or you know whatever. I don't remember the exact sequence. You could almost envision like this giant herd of bison like thundering from one little search watering hole to another. And then they would hear that there's even more water available at a different search hole, and they all go thundering off. I mean, did Google kill that basically?

Danny

The movement from one place to another and so on?

Eric

Right.

Danny

A little bit. People would do that sort of thing, especially in the early ages when...I started with Lycos and Yahoo. I mentioned, I was on the web, and then I immediately had these two different kinds of tools. And then you would get these other ones that were all springing up, and you're right people would go, "Well, let's try a sample here, let's try a sample there." And they all kind of ended evening up with the kind of share that they had. And then what you had was a really, really big shift when Alta Vista came along because it was so much better for a lot of searches that people, a lot of them, especially those who were seriously searching, moved over there. And so if you will, some of the other watering holes started drying up and Alta Vista kind of hung in there. I think then the last great migration that we had really was the shift from Alta Vista to Google. And there are two things that I especially noted. I used to track the search marketing shares over time, and there was what I call the Google-Alta Vista X, which was you could watch the number of people abandoning Alta Vista and moving over to Google, and it was like an X in terms of the trend lines of one going up and one going down at almost the exact same proportion. And the other thing that was more anecdotal. If you ask people who, and you still find this happens sometimes, you'd say to people, "What search engine do you use?" and for the longest time, 2003, 2002, 2001, maybe the 2000 [inaudible 00:34:24], I would very commonly get a response from people like, "Well I used to use Alta Vista, but now I use Google." And it was almost like talking to someone who was divorced, or had broken up with someone and wasn't necessarily happy that that had happened, "Well, I used to be married, but it didn't work," or, "oh, I was going out with this girl, but yada yada yada." People liked Alta Vista, they almost begrudgingly left Alta Vista in the end because Google was a much better experience, a much better search engine for them, and they still had this love and a lot of affection left over for Alta Vista, but it just couldn't get there. You don't really hear that these days because the other big shift that I started encountering, and was absolutely strange to me, was people started talking about "googling" stuff. I've been writing about search engines for so long, and I'd always referred to them as "search engines" and "searching" and then people started talking about how, "I googled this," or "I googled that," and the idea that a search got turned from a generic name, got turned into a "kleenex," or that the brand was going to actually represent the process that you do, that search became Xerox, that type of thing. That was a shift I was finding from people, I think that really just seemed to be coming in, where either they got onto the web a bit later, or just really was reflecting that they didn't even know that there were other watering holes. It really was just Google or nothing and they've been doing it all.

Eric

[00:35:58] So, can you recall what were the actual experience differentiators between Alta Vista and Google? Because we know that Google had a better algorithm. Was it just that you got your result faster, or was there more to it that sort of drove people to, as you say begrudgingly, move from one to the other. Because I don't remember exactly my own experience, I just remember that sort of begrudging feeling of "Well, you know Alta Vista, yeah, but Google is so much better now."

Danny

Well you know Google, first of all it was the relevancy, it was that even though they had fewer pages than Alta Vista at first, they were giving you better quality answers because they had come in with this system of looking at how people were linking and analyzing all the links and kind of considering them both as votes and also analyzing which votes were important and what the votes were referring to, and that relevancy is what set them apart. And then they started increasing their comprehensiveness by adding many many more pages so that you were even finding the obscure answers that were out there. And then it was just this relentless improvement of features that they initially started mirroring all the features and matching all the stuff that the competition had, and then they went beyond it. These days, there's a thing that you can watch on Google where they talk about what it was like to be in one of their search quality meetings, this was when all the people involved in search get together. And I think it's fascinating first of all, just to watch this process of how it all happened, you understand some things they discuss. But they discuss things like making sure autocorrection is working for three word queries in Vietnamese, right? These really incredibly granular things, and that just, it illustrates just how far down, or how specific they've gotten to all these little improvements, and I think all those sorts of things add up.

Eric

Yeah.

Danny

And the competition just really wasn't making that kind of investment. Alta Vista, in contrast, belatedly decided they should be a portal. And so their money was going in the wrong place, the talent is going in the other direction, so it's a big change.

Eric

[00:38:29] Yeah. Is Google doing that though? Are they becoming a portal? Or are they trying to be? Because they keep adding stuff, but they never seem to quite tie it all together. It's a bunch of stuff.

Danny

Well they are. The time I wrote a thing once where I referred to them as portal 2.0. What I was saying was that the new portal, or the stealth portal, is the portal you don't think of. That they were adding all these kinds of features, but not necessarily trying to force you to be unified through a homepage. And they didn't necessarily need to do that. It allowed them to compete with some of the portal players, like Yahoo, which had email, they came up with gmail, and it allowed them to compete in that space as well and pick up some of those people, but do it in a different kind of way. So, what's really interesting now is of course the portal 3.0 if you will or whatnot, which would be more of your Facebook situation, which is people wanting to start their days with a social network perhaps, and getting their news through what's being shared by their fiends or what's in their newsfeed, as opposed to having to go to a page where it's all being laid out. And that's definitely a challenge that I think Google is facing now. They don't really have that kind of starting place, that's what Google + is all supposed to be about, but more, because google + is also supposed to be about just sharing throughout Google itself and making Google's experience more social. So we're watching how they do that. But I think they've had great gains, and people talk about the activity that's going on there and how it all works, and they're well behind where Facebook is at. But they do have some significant people that are there, so it's been an interesting pivot, to use the popular word these days that they've kind of made in that regard.

Eric

[00:40:26] Yeah, I'm wondering how much of the verbing of google to steal a phrase, to steal a pattern from Bill Watterson, how much of that was driven by putting search into browsers, do you think? Like making it part of the browser chrome so that you didn't have to go to google.com in order to search. Did that contribute really to the verbing of the term Google or had that already happened and it was just an after effect?

Danny

I think that's definitely helped, and I think that's been engendered a lot out of Google's absolute paranoia that they would get cut off. It's, I think, a deeply paranoid company that has always assumed that Microsoft would somehow drop it out of the operating system. Therefore, "Our answer will be, 'we'll create our own operating system,' Chrome OS," and, "We're paranoid that people will not be able to get to us through Internet Explorer because Microsoft is a deeply evil company, therefore, we'll create our own browser," which by the way squashed the efforts of people using Firefox. A lot of European people use Firefox, but Chrome definitely pulled away from the dominance that Firefox once had. And then, "Oh, Apple's getting really popular on phones? Well, what if, despite the fact that we actually have a deep partnership with Apple to the degree that our CEO is on Apple's board, let's start our own mobile operating system, because that way we can ensure our survival with search and we won't get cut out." And so all those things definitely have driven tons and tons of queries into Google in ways without people having to go to those webpages--into the homepage and doing the searches. And that's only going to continue, that searches are not just you go to google.com and you type in words. And Google has positioned itself well to try to be in other places for that to happen.

Eric

Yeah. Of course they generate tons and tons and tons of revenue off of all those queries they get directed into them, one way or the other. It's interesting, I wanted to go back because you mentioned when Open Text said that they were going to have paid results basically, and the internet went crazy, I presume because that was, "Well, they'll bias the results and in the direction of the people who paid them," of course, because that's the point, but that will cheapen it. Do you feel that Google sort of avoided that because they were a third generation search engine and people had time to get used to the idea, or did they do it in some unique and more clever way?

Danny

Take nothing away from the fact that they're incredibly clever and they spent a lot of time improving search and improving ads and doing all sorts of other things such as bring in ad sense which fueled some of the other stuff that was going on in terms of their revenue stream. But yeah, I think it may have been a much different story if they had started in 1996. I'm not saying it wouldn't have happened, it's just that at that point, it may have made sense for them to do other stuff. I don't know that they would have necessarily thought, "Well, let's go through, it didn't work for Open Text, so let's just say forget it, we'll pick up the paid ads right way." If you go back to their early papers, even they were dubious about whether or not paid ads would have worked and how that would have come together. I still think they would have gotten there, but it potentially would have been a different situation than the one we're looking at now.

Eric

Yeah, that's kind of funny that the early papers by Larry and Sergey would be dubious about the value of paid ads and now they both own their own private 737s as a result of them.

Danny

Well, it wasn't that they were dubious about the paid ads, they were concerned about the conflict you had between ads and search results and just presenting the search results. They felt like it presented this conflict that was just going to ruin the situation. How do you resolve that? And apparently they think that they've resolved it because...I don't know. In an ideal search engine, you never click on an ad because every time you do the search, it's giving you the exact right answer. And if they really are the greatest search engine in the world, as they aspire to be, then eventually they should put themselves out of business because there's no need for people to advertise because they already gave you the right answer. The reality is that search engines are imperfect creatures, they cannot make those kinds of perfect things, so where they make their money is off the impreciseness. They make their money off of being imperfect. And they're perfect most of the time, and in the times when they're not, they figure, "Well, ok, I guess we'll cash in on it a little bit here or there". It's not as crass as that, I don't they they would ever express it that way, but that's the reality of what happens. And that's the conflict, because if you decide, "Ehhh, let's be a little more imperfect in this area, let's be a little more imperfect in that area," you could do it deliberately and potentially make more money, or you could just decide that, "we've got a whole bunch of things that we could try to be imperfect on, let's focus on the things that we think matter, let's try to be perfect when people search for suicide prevention, let's try to be perfect when people try searching for health information. People are searching for online gambling or mesothelioma warriors, I don't care if we're that perfect, so let the ads duke 'em out."

Eric

Huh. Interesting. That, huh. I'd never thought of it that way.

Eric

[00:49:38] As we were talking I did recall actually, as the brain slowly spools up, one of the things that really hit me about Google that eventually made me switch over was that what you were looking for was what you were looking for was almost always on the first page of results. Whereas on Alta Vista, Lycos, or some of the other ones, you'd have to go a few pages before you found what you were looking for usually. At least at the time, in the '90s. I assume that's just that's the PageRank was a much better algorithm, they had a better recipe.

Danny

Yeah.

Eric

But there was that period where you would do a search for something, and you knew you were going to click through some pages, I don't think it would have occurred to anyone to think that that would not happen, right? You were going to have to go five or six or seven pages of ten results per page before you finally found that thing you were looking for. And if it came up on the first page, you would tell your friends like, "Oh my god! I was on Alta Vista and I searched for 'css support in netscape navigator' and the third result was exactly what I was looking for, I couldn't believe it!

Danny

Right.

Eric

Whereas now, does anybody ever hit the second page of Google results anymore? Really? Unless you're searching for something incredibly obscure.

Danny

Yeah, it's rare, and in fact it was already rare with any search engine. People just don't like to go past it. If they kept going past it over and over it seemed to build into a fatigue where they'd be like, "Ok, I'll just use another search engine." And that is still the case, they are very, very focused on how do we ensure that we have the best results in the first page that's coming up. I think the stats are something like 90% of the people won't go past it. And that really means that they find something that's there. You are giving them ten results so they probably will find something that's sort of a match. If you're consistently causing your searcher to keep going into the second page of results, you might not have searchers for very long.

Eric

[00:51:44] Yeah, seriously. Because how many people even bookmark anymore? That's to me one of the fundamental changes that happened, and I feel like Google was the search engine that caused that. Or they were there at that time. It used to be people would publish their bookmarks. Netscape Navigator at one point made it really easy to export your bookmarks as an html file so you could just put it up on the web. And I think there were even plugins or maybe Netscape did this natively, where you could autopublish your bookmarks, because if you had a really great bookmark collection, that was it's own mini-directory. It was your very specific, categorized-the-way-that-it-made-sense-to-you directory, but it was still a directory. Bookmarking was really important. You found something that was useful? You bookmarked it because you might not ever be able to find it again, right?

Danny

Yeah.

Eric

You'd remember that. And nobody does that anymore because all you have to do is remember in your head sort of what were the four words that you searched to find the thing the first time. If you searched for those four words again, you will find it again.

Danny

Right.

Eric

That's been a huge shift for me. There are a lot of people, like we said, who will be listening to this, who'll not remember any other way. Some of them might not realize consciously that there's a bookmarks menu in their browser, and even if they do, they probably never use it. That was a fascinating shift, to me. I felt the same way about Google that I think you described, which was, had they come at a different time, we might be talking about...like, Lycos might be a verb, like, "I lycosed a thing," right?

Danny

Sure.

Eric

Had the timing been different.

Danny

[00:53:39] Yeah, it's possible. You could certainly play the 'what ifs'.

Eric

Yeah, exactly. We don't have the quantum tunneler to look over to the next universe and see if Google won over there as well. Although I'm sure they're working on it, somewhere. Google, themselves, working the quantum tunneler. Because, you know, driverless cars, that was an obvious outgrowth of search, wasn't it?

Danny

Exactly. But see, that's Google's paranoia again, because they're worried that people will be driving and not searching, therefore.

Eric

[laugh] I was going to say! I've seen people make the argument that, "Of course Google is working on cars that you don't have to drive because then you'll be on Google more. When you're in the car, you'll be looking for stuff."

Danny

Right! Well, it's crazy, but I think a part of this is also because you've got these Google founders who really do love big ideas, but you're in the car, the car can take you anywhere, and you're hungry, so you say to the car, "Car, I want to go to get something to eat," and it brings up the search results, and you click on the search results and the car drives you there. That's a search. That is literally search built into your car. When you think about the things we do with the cars that we own, most of the time it's because we're doing some kind of a transaction, or a lot of the time we're going to do something like that. So the idea that you have a driverless car that also has the search built into it, and you can find what you're looking for and you can click on it, it takes you right where you're going to be going, you sit back. Or that you're in the car and you're hungry and the car's driving you some place, and you're saying to the car, "Car, I'm hungry, where should I eat?" and it gives you ten listings, and because the car knows where you're at, and the advertisers know where you're at, the advertisers saying, "hey, why don't you pop into Denny's or Big Red House," or whatever it is, "because we're having a special offer and you've never been here before because I know exactly who you are because Google tells me who you are because you've got a registered car. And we'll give you a free meal." "Alright, let's go try that!"

Eric

Wow.

Danny

And cars aren't going to buy themselves!

Eric

Sure.

Danny

That's where some of this stuff potentially ends up.

Eric

Wow. Well, we've just looked from the past to the future, but that's kinda fascinating that it all grows out of that.

Danny

Wait till we get the jetpacks.

Eric

You know, advertisements while I'm jetpacking? I don't know if I like this.

Danny

Yeah. [00:56:12 inaudible]

Eric

[00:56:12] We might need to draw a couple of lines somewhere. Anyway. Before we wrap up, as you look back, are we seeing the same patterns happen here that we did before? Or alternatively, are there some major differences between now and then that you see that we haven't really talked about so far?

Danny

What's been interesting is whether or not...the patterns the search engines had in the past was that they took their eye off of search, and in doing that they got overtaken by another player. And that is one of those patterns that I sometimes wonder if Google's going to do themselves, as they do everything else, and sometimes they have these little missteps that I don't think the typical public sees, but I'll see, or search watchers will spot. Like when, I wrote a thing on Friday about how they had introduced a feature that allows you to block search results. They did this over a year ago. They said, "Hey, you don't like somebody's website? You think it sucks and you're tired of seeing it all the time? Here's a feature that lets you block it." Well, that feature got dropped last year, they rolled it out two years ago, it got dropped last year when they made another change, and it's never been brought back. And they didn't bring it back, even though the help page still says it's out there. And it's one of those things that got lost in the shuffle. And they were doing so many things that are going on there that I feel like sometimes, are you losing track of some of these other important stuff that's going on? At the same time, they are being very nimble in trying to respond to the whole social situation, and with the idea that the social data might very well be useful data for search, and so effectively building, yet again through paranoia, their own social network. I talked to you about what they did with mobile, what they did with the OS, whatever. So they look at something like Facebook and they think, "Wow, we can't get this data, it's all locked up in Facebook and we might need it, what do we do?" And for a while they were working with Twitter, until that falls out, and that kind of almost fuels their paranoia that, "We can't work with anybody, we need to build our own independent social network," and hence you get Google+ that comes up with it from there. But they do seem to perhaps be responding to these challenges as they come up, as opposed to being stuck in the mud and not making changes at all. And I think on the whole, they've been much more nimble than a company their size you might expect.

Eric

[00:58:52] Hmmm, interesting. So last thing I guess. Metatags and stuff like that. They used to be really huge. You had to have the right meta keywords and the right whatever. "What used to be important that isn't anymore," is how I'm going to frame that question. If anything? In terms of the markup and the structure, the sorta technology side. What has become unimportant over time?

Danny

Well, interestingly, Google, who had never supported metatags, actually brought them back in for their news site, and in fact a lot of the structured language that they brought in to try to get people to use stuff that's related to schema.org is metatagging in another way. So that stuff's actually been resurrected, just in a different fashion. I guess the tried and true stuff that doesn't change is: having great content that's in text readable format, is the best I can put it. You've got great content, it is not all locked up inside images, it's actually cut and paste style content, and it's descriptive, it's using the kinds of words people are searching for. Those are things are really important and continue to be really important. Beyond that, there's just this constant evolution of new features and options that you can have. Everything from being able to link your pages so that they have an identified author to being able to feed out an actual XML site map list of all of your content that's out there. And in fact there's so much of it, it changes so much, that I think that it's difficult from a design standpoint, for a designer, to keep track of it all, anymore that it is possible, say, for someone who's a search specialist to think, "Ok, I'm going to know all the design stuff that's going on out there." I think that there's a lot of best practices that a lot of people can get down right, but it also helps if you're worked with people, or you can keep people to keep you on top of the stuff that's really important. And there's degrees in it as well. The good news is, if you've created a good website with the stuff I talked about, the tried and true stuff, even if you don't do anything else, you'll probably get some decent search traffic coming to you. And the other big thing that's out of your factor that's hard to control is the inbound linking, which is still very, very important. But that goes back to having good content. If you've got the good content a lot of times that attracts links. The biggest and most significant change that has evolved in that aspect is the whole social layer. That if you've got the good content and you've built out the social accounts, and you can spread it that way, that that's a good way and fairly easy way of building out some of the links you might need.

Eric

Mmmm, that to me that's like Webring 4.0.

Danny

What's that?

Eric

Webrings? Remember Webrings?

Danny

Yes.

Eric

Yeah, so the social network is like Webrings 4.0. Or something. They're so far beyond. But it's the same kind of thing, right? Because webrings were a way of associating a bunch of topic related sites together. So you would have the Babylon 5 webring, which was 50 or however many sites, all done by different people, but it was like, "Oh, you have a Babylon 5 site? I have a Babylon 5 site? Let's link them together!" And then you can click from one to another and that was a way of associating things and creating these mini social networks before we had social networks. Did web crawlers ever give weight to webrings? Or was that: a webring is a link, it was part of a pattern?

Danny

I didn't catch that last part. Did web crawelers...?

Eric

Did web crawlers give extra weight to webring links, or were they just considered to be links like any other.

Danny

No, it's difficult when you start talking about links. They have all sorts of ways of weighting them up. Webrings were a way that people could link together, and that might help, and then down the line the search engines might realize these links don't have a lot of value. They don't want to just reward anybody who gets a link from anybody else just because they give it out as well, so they do try to figure out which ones are important or harder ones to earn.

Eric

Oh, interesting. Ok, well, this has been a lot of fun, we could probably go on for a while but I think we're just about out of time, so I'd like to thank Danny Sullivan for joining us. Again, if you want to see what Danny's up to, you can checkout Search Engine Land, or Marketing Land, the Common Sense tech column at cnet, or the Search Marketing Expo serise of conferences that he's got going. Danny, thanks so much for joining us.

Danny

Thank you for having me. I appreciate it.

Jen

Dan, people can follow you on Twitter, can't they? If they want to keep up with you?

Danny

Yeah, it's just @dannysullivan, my name at symbol. At twitter.

Jen

And Eric is @meyerweb.

Danny

[Inaudible]

Jen

That's alright. @meyerweb is Eric Meyer, I'm @jensimmons, the show is @thewebahead, all on Twitter. There's show notes for this show, where we've collected links, you can go to 5by5.tv/webahead/48 because this is the 48th episode, and I want to say thank you so much to our sponsors, Event Apart and the Chrome Channel on Google Developers Live for supporting the show. Thanks everybody!