Jess McMullin: This podcast sponsored by the American Society for Information Science and Technology. Since 1937, ASIS&T has been the society for information professionals, leading the search for new and better theories, techniques, and technologies to improve access to information by the IA Summit. This year your peers and industry experts spoke about how topics such as social networking, gaming, patterns, tagging, taxonomies, and a wide range of IA tools and techniques help users experience information and buy Boxes and Arrows. Since 2001, Boxes and Arrows has been a peer written journal promoting contributors who want to provoke thinking, push limits, and teach a few things along the way. For other events happening all over the world, be sure and check out events.boxesandarrows.com.
At the 2008 IA Summit in Miami, Florida, Jared Spool gives an enlightening and entertaining keynote address entitled, Journey to the Center of Design. There's a growing sentiment that spending limited resources on user research takes away from the essential design activities. Previously, fundamental techniques such as usability testing and persona development are now regularly under attack. Let's not forget that today's shining stars such as Google, Facebook, Twitter, and the iPod, came to their success without UCD practices. I hope everyone enjoys the keynote address from the Summit. Cheers.
Jared Spool: What I want to talk about ... When I was a kid, one of my favorite movies they used to show every other Saturday on WSBK, a little Boston television station, which I don't know what they show now because we don't watch television, but then when I was 14 years old they just showed movies all weekend long, old movies. They used to show this one with James [Mason 00:01:50] was Journey to the Center of the Earth. I thought, we want to talk about design, how we got to the center of design, where the core of that is. I thought we would talk about that a little today. Of course I'm thinking, I've spoken at a lot of IA Summits. This is actually my second IA Summit keynote, which is really interesting. I think you've heard enough of me at some point, so I'm thinking that everybody else just said, "No," at that point.
The first IA Summit I talked about this idea of sense and how people find things and it was very interesting. Back in 2001, your IA Summit was just beginning, this is the second year of the Summit, and it was really focused on how do people practically use large information systems, so we talked about sending information. Then a couple of years later I did a talk on some research we did involving different types of information users look for, and after that I did another talk about the research we'd been doing in the psychology of decision making and how different types of pages on a website support or don't support different types of decision making. A couple years ago, I talked about the difference between how do you instruct teens, and talk about the difference between specialists and generalists, and talked a bit about why you don't get to choose whether you're a specialist or not, but in fact, the economics of wherever you work chooses that.
Last year I talked about, I don't know, some web 2.0 crap, which turned out to be completely work. Eh, it's research. All that was really practical and useful, today I'm not talking about anything practical or useful. Because with a keynote, you get to pontificate it, so I though I would pontificate that. There'll be some mildly useful bits in the middle. I'm hoping there's going to be some entertaining stuff. I actually bought my knitting, so we're going to play with that. We're going to talk about stones, and we're going to talk about why AT.com really sucks, just to name a few. I was recently at the CHI Conference in Florence and one of the comments that came in on the CHI evaluation form, I happened to see, was that CHI, which is the Computer Human Interaction conference, is a fairly academic conference, there's lots of research papers that are presented.
One of the people who attended wrote on their form that they weren't as happy for this year's presentation because there wasn't enough blood. I thought we'd start with a celebrity death match. The celebrities I've chosen, I've decided for today is the folks at 37 signals versus Don Norman. This isn't just a random pairing, these 2 actually paired themselves up all for me. You see, a few weeks ago Wired magazine came out with an article about 37 signals, and it was this, "These are the brash kids from Chicago who are defying all the rules of business and building great products," and all this hero worship stuff. It was a very interesting article, and all of the sudden there was this real controversy, or as Richard would say controversy, that came from this. The folks at 37 signals said in the article, they said, "We're not designing for others, we're designing for ourselves."
If you've paid any attention to 37 Signals and their products, basically [every 00:06:06] campfire stuff, this comes out so much in their work. They wouldn't put a calendar into their base camp product for years, because they just didn't understand why you'd want it. Even though users kept asking for it over and over again, they just didn't do it. Don Norman read this article and his response to this was, "I've tried their products, and although they have admirable qualities, they never quite met my needs. Close is not good enough. After reading the article I understand why the developers are arrogant and completely unsympathetic to people who use their products." Now, I've got to tell you, Don Norman calling someone arrogant to begin with is an interesting thing. Mr. Kelly, we [inaudible 00:06:57]. That in itself got my attention, but what was really interesting about this was Don is coming from this, from being one of the people who was in this field before I was in this field.
Don came out of this with having worked with CSB from the world of ... Don't mind Stacy, she just randomly feels the need to nest. Don comes to this from this heavy academic, psychological space, and he really was one of the people who started talking about how we can't just think about designing for ourselves. We have to think about the people we're designing for and how to design for them, so it makes sense that he would pick up on this and he would really go for it. The 37 Signals people have actually been really popular, really successful, and it's very interesting because they don't do that. They don't do what Don says they should do. The thing is they're not the only ones. Over the last few years Apple has been, piece by piece, shutting down their user research organization. The iPod was created with virtually no user research. Apparently I just [inaudible 00:08:31] on Second Life. The only thing I've seen people do on Second Life, I'm not going to do.
They've been very successful. Apple has been shutting down their user research labs. The iPhone came out with no formal [user delay 00:08:56] testing, no formal user research. There is this movement afoot that says maybe this idea of designing with users in the center is not the only approach, it's not the only way to do things, and maybe there's another way to think about the problem. There's another way to think about how this is going to work and so to talk about this, I thought I'd go into a little bit of history and talk about where this notion of user centered design first came from. If we step into the time machine and go back, it came from this. This is the IBM 360 mainframe. Just out of curiosity, how many people here actually worked on one of these? I love speaking at these sorts of conferences because people here are awake. I do a lot of talking at Microsoft, where the average age is 14. Most of them are prepubescent, like the product managers. It's nice to talk about it. Now, [inaudible 00:09:59].
This system actually came out in 1964, but this version is from 1972. How many people weren't born yet? God, that's depressing. I hate the idea that I have to work on equipment that's older than you. The IBM 360 was an interesting device. The one thing I want to draw your attention to, this big red button right here. That big red button shut down the whole thing. You press that, boom, it shuts it down. If you were to use it [inaudible 00:10:40] like AARPs require. If you press that button, I remember this because I got this lecture on the first day, don't press the button unless something really bad is happening, because it takes a day and a half to get a technician in to undo whatever that button does. It shuts down everything and everything has to be redone. The big red button was a problem.
The thing about this device that I wanted to talk about, was that this was developed by engineers, for engineers. The people who used this thing were not normal people. They were a group and they were very highly skilled, they had to have a certain way of thinking about the world to be able to operate one of these things. They were also very highly trained, and when they were using it, they were focused completely on it. Their day was focused on making this thing work, serving this master, that's what they did, that was their focus.
Now back in 1980, IBM worked on a different product, they worked on this, it's the display writer. This is a $15,000 word processing unit. All it did was word processing ... Look at the size of that keyboard, it weighed 32 pounds. A keyboard, just a keyboard. This thing was not designed for engineers, it was designed for office workers. What was interesting about this was that you had take a completely different approach, you had to think about ... The people who are using this were not skilled in using this device, they were secretaries, and stenographers, and office assistants. People who were skilled, but they were not skilled in technology by any stretch of the imagination, nor were they really trained with it.
To use this device you had to go to classes and you would learn how to save a file, load a file, change derivative and all these things. This was not the extensive training that the engineers received which was multi-year training. This was a few weeks in class and hopefully that would do the trick. Finally, these people could care less about the device, they only cared about the data that they were working on in the device and the context that that data was being operated in. It was a completely different thing and the techniques we used when we were designing stuff for engineers don't translate for designing for people who aren't focusing on the tool, who aren't going to be highly trained.
That's where this notion of user centered design was, we actually had to think about, well gee, what does the user know, what do they come to the table with, how do they use it, how do they think about it? No one had thought about this before, no one had ... If you ever saw the movie Apollo 13 where they're in the control room and they're trying to figure out how much air they have left, and everyone whips out their [Springboard 00:13:55] and starts working with it, they all knew what a [Springboard 00:14:01] was, they all knew how to use one. These people did not, these people were a completely different breed of folks. Other than marrying them, they didn't know how to relate to them in any way possible. Designing for these people, we had to come up with a whole new process for coming up with software and hardware, this is where user centered design was born.
It was really in this reaction, moving from the space where we were doing things only for highly skilled folks where we could let things slip through the crack and they would read the manual, they would understand how to do this stuff, and they would go through the work, and start to work with these non-engineers. Of course all of the original [inaudible 00:14:51], interestingly enough it didn't come out of the world of engineering or design, it came out of the world of psychology. The first people in this field were all trained psychologists. They were all ... Psychology was all focused at the time, on, human capability. Human memory, human movement, all these things, so it was a natural outcome of that. There was a human factors part of the field too. Human factors started about 50 years before, in corporate industrial efficiency and was working in that. It didn't really come under it's own right at about the same time, when nuclear power plants started melting down out by Harrisburg, Pennsylvania.
The Department of Energy showed up at the annual meeting of the Human Factor Society, which had 264 people in attendance that year, and said, "After learning that human factors played a role in the 3 mile island meltdown, we decided we need to hire new human factors people, where can we find 3000?"
The human factors field really blossomed at that point, and all of the sudden, we saw this influx. It wasn't from a process of delivering products or or delivering software, it was out of this psychological, emotion study stuff that it came from. It had a promise with it, and it was really popular [inaudible 00:16:15]. We used to go around talking about how everything's going to be user friendly. That was term that everything was used, user friendly. It got so overused, that it was like "New and improved on a box of pie." Everything could be user friendly and the promise was, that if you make it user friendly, if you create an extension with the users at the center of the design process, then you would have better market acceptance, you would sell more product, people would love you more, they would ...
That brings me to the elephant in the room, the problem with this idea, the problem behind all this, is that, this idea of user centered design, it's never worked. It never has actually produced anything. That's what's really interesting, we can't point to any instance where this was the key variable to success. We've been doing it for 30 years, and we can't do it. My company, User Interface Engineering, we're a research company. Technically we're a consulting company, we don't [inaudible 00:17:27] and we don't like doing that, so we found the less we do the more we can charge, so we just do research. We've been researching for the past 5 years or so, and I've recorded some of the results there.
We've been researching what it takes to create great designs. If your company wanted to come out with the next iPod, or Netflix, or Nintendo Wii of your industry, what would it take how would you hire, what would it be. We've been doing all this research, and one of the great questions is this question here, how do the best games create great design? To share what we've learned from research, I want to talk about some of the stuff we found. We've been going and talking to different teams. All sorts of different teams, and one of the first things we do is we assess where on the scale they are on terms of producing great experiences. Are they a team that regularly can come out with products that people love that have great experiences, or are they a team that tries to do that but really struggles. That's the spectrum that we're looking at, from people who succeed, to people who really struggle, and want to do it, but they struggle.
As we've been doing this, we've been looking at the way that they work and how they do their projects. One of the things we found early on, was that there's a way to think about how you get things done, and at the center of this way to think about how you get things done, is this idea of process. Process is a [inaudible 00:19:09] term, it's very overburdened, but we have used process in a very particular way. It is the steps required to accomplish something. Sometimes you hear people going through their organization and they'll say "Our big problem is we don't have a process." They deliver things, they ship things, so they must have had a process to get those things out. It might not be a process that they could repeat, it might not be a process that they'd ever want to repeat, but they had a process.
Think of a process, like a recipe for cooking something. My mother is an amazing cook, she's an absolutely stunning cook, a master chef. She's of Hungarian descent, so she has all these great old Hungarian recipes. One of her favorites, one of my favorites, is her chicken paprikash. For years I've been asking her for the recipe for her chicken paprikash and she tells me every time, "I don't have a recipe, I just do it. I've been making this since I was 14 years old, I just make it." I say "Well, you must have a recipe, you make it." She says "No, no I just make it, I don't have a recipe." I do it different every time she says. I bet you if i stood next to her when she made it one time, and I wrote down every ingredient she put in, and I wrote down the order she put it in, and what she did every step, when I was done, I would have a recipe.
She'd be the first to tell me that she'd never make it that way again, and that's okay because I would now have the recipe. That's the idea right? That's a process, a process can be a one time instance, it doesn't have to be repeated and it's okay. You have to have a process if you got it done, that's the way it works. What people confused process for is actually this other thing which we call methodology.
Methodology is the ability to do the same thing over and over. Organizations love methodologies, because the idea that if they do something well, they should be able to just do it again, [at-the-same-time 00:21:14], that's a brilliant thing. Methodologies actually have their own little scale where they go from [inaudible 00:21:19] methodologies where people pick up a habit and do it over and over again, to formalized methodologies, where you write everything down, you put it in a special notebook, and give it a funny name, [inaudible 00:21:29] money for it. You've got the scale of methodologies, that work, and then there's this thing at the far end of the scale, which we call, dogma.
Dogma, let me explain dogma, the best way to explain dogma is to actually talk about the government organization which is most famous right now for their dogma, and that would be the Transportation Security Authority. These guys are specialists in dogma. TSA Formally stands for Transportation Security Administration, though some people suggest it stands for Thousands Standing Around. Has created this entire way of thinking about getting to the airport. Which is new and novel and filled with adventure. One of the pieces of adventure has to do with the idea of the 311. If you don't live in Miami then you've experienced the 311. The 311 says you're allowed to 3 ounce things, stick them in a single one quart bag, that's 311. If you can't fulfill that need, if something is more than 3 ounces or it doesn't fit into your single 3 ounce zip-lock bag, then it must be dangerous material. Of course, the TSA looking out for the good of all people, take these dangerous hazardous materials and they safely store them where no one will get hurt if in case, they explode. This of course is safe, in their mind.
The idea of this is actually very ... I travel a lot, about 25 conferences a year, I'm always on airplanes, I'm always going through security, I've gotten to see some things over time. For instance one of the things I learned is that cream cheese, a little container of cream cheese that you can buy at the shop right outside the TSA, this is something you can take through security, interestingly enough. They allow this, they don't even require you to put it in a little bag or something. It might be, if you look real close, the label of this cream cheese has the word "real" in quotes, so it might be ... So cream cheese is good, but yogurt, this is not so good. You're not allowed to take yogurt through security. It's a gel, apparently cream cheese is not a gel, I don't know what it is, but it is not a gel, it's cream cheese. Yogurt is a gel you can't take it with you. They decided that this is safe, but this is bad. We see this all the time here, this is cream cheese, it's safe. This is yogurt, it's bad. Bad things explode, we don't want terrorists to have yogurt. Cream cheese they can have.
I've always ... So one day I was going through security and there was this really cute woman in the line in front me. She put her stuff through the machine and she gets to the other side, I put my stuff through the machine and wait for it to get to the other side. When I get to the other side of the machine, the woman is having this discussion with the TSA about this one thing of lotion that she has, single tube of lotion that isn't in a plastic bag. It has to be in a plastic bag or she can't take it on the plane. Apparently it's a valuable thing of lotion she doesn't want to just throw it out and she says to the guy, "Do you have any bags?" He says, "No we don't have any bags here, but there are bags at the other security station at down at the other side of the airport." She's like "Well I have to get my flight, I can't go down to security, grab a bag and come back." He says, "I'm sorry you have to put it in a bag." She says, "Does anybody have a bag?" I said, "Well I have a bag." She said "Oh that's so nice of you."
So I just come through security, I paid for the things out of my bag, and as I'm doing that the guy looks at me and says "What are you doing?" I said, "I'm giving her my bag." "You can't do that." I said, "Why not?" He says, "Because it's not safe." So you just x-rayed the bag, you know what the bag's contents are, you've proven that its under the limit. "Well what are you going to do with your stuff?" "I'll put it in my suitcase." "No you can't do that." "Why not?" He says, "What about the spot checks?" I spend so much of my life in airports, I have not seen a single spot check, but the threat is there, it could be there any day, what about the spot checks? This is the deal right? Lotion in the bag, this is not a terrorist instrument. Lotion outside the bag, this is a terrorist instrument. Somehow, a terrorist can't deal with zip-lock bags.
This is dogma, dogma is the unquestioned faith, that something actually is important without any evidence to suggest that that might be the case. To me, TSA is security theory. There's nothing safe about it, if somebody really wanted to get stuff through, I'm sure they could figure out a million ways to do it. It's all about looking safe, it's all about feeling safe, you're not allowed to question the rules, there is no discussion about it. In fact, the rules are hardly published at all, and in fact, people have taken the TSA to court for things like unwarranted search and seizure, they say "No we have complete authority to do this." Well show me, "I'm sorry that's classified we can't show you." It's just this whole mask of stuff, and that's dogma. We have the same thing in or field, we have this unquestioned belief that there are things that must work, and they must be the way we do things, and we just follow them blindly without even questioning them. It's the same sort of [theater 00:28:32]. It's the same sort of thing.
As we've been doing our research, we found that ... What we were thinking ... So we go out and we talk to these teams and we assess, what is their process and methodology, we're trying to figure out what these things are. What we were thinking when we went into the research 5 years ago, was that what we would find is that those organizations that are extremely successful at getting great experiences out have discovered some methodology or some dogma that actually works for them, and actually succeeds. So we wanted to see if that hypothesis was correct. What we found was that we need to pay attention to something else, and what we needed to pay attention to was the stuff that happens on the other side of the toss-ups.
That starts with this idea of techniques, now I said a process is a series of steps to get things done, the techniques are the things that go to each step. One or more techniques is part of each step. There are little independent things, for example when you cook, when you cook chicken paprikash. One of the things you have to do is you have to make a roux. Now for those of you who have never made a roux, it's actually very simple. You just take some flour and some milk and put it over a little heat and you stir it. It's not so simple, because getting the heat right or wrong is really important. The rate at which you stir is really important. There are some real practical elements that you can only get by just doing it over and over again. The nice thing about a roux, is that the ingredients flour and milk are so cheap that you can screw it up 5 times and still get your meal done. You sit there and you keep doing it, if you have it too hot or you don't stir it enough it burns. If you cook it too low heat it doesn't quite cook right. You have to find the exact perfect balance, and that just takes practice.
That's a technique, once you've mastered that technique, you don't just use it for chicken paprikash. You use it for desserts, you use it for, soups, you use it for gravy, you use it for all different parts of the meal, and you use it independent of the cuisine. If one wants to think of cuisines as methodologies, Mexican food, French food, German food, Hungarian food, Chinese food, they all have roux in them. This is just an independent technique that once you've mastered it you can use it a million different ways, and techniques turn out to be really important. On the far end of the spectrum there's something else, we couldn't figure out a name for it so we just called it, tricks. Tricks are techniques that you use, but not for the purpose they were intended, for something completely different.
The best way I have to explain it is, a few months ago I needed to call a plumber. It occurred to me when I was trying to find a plumber, that I never once called a plumber and asked them what methodology they follow. I don't know what a plumber would say if I asked them what methodology they follow. "Well we've been using a waterfall model ... " Where we had the inputs and then the outputs and we trapped the [flows 00:31:45] between them. Lately we've been moving our organization more to an [object-oriented 00:31:46] model. Where we treat every drip as its own object and we send it a message that says, fix yourself. What would a plumber say, I don't know.
All I know is this, I hired the plumber, the plumber comes to my house, he drives up into the driveway, I'm very excited on this. I go run into the driveway and he opens up the back of his truck, and has all this stuff in it, it looks like it had just come straight from the Spanish inquisition. He put some of them into a toolbox like thing, he brought them in the house, he went into the basement, he looked up at the pipe, he rummaged through the toolbox for something he ... He picked it up and started banging on the pipe.
I said to him, "How did you know, without seeing the problem, how did you know you were going to need that tool." He said, "Well, I didn't really." Then I asked a question I shouldn't have asked, "Is that the right tool to use for this?" He said, "No." Paid this guy like $700 an hour and "No." So next stop, "Why aren't you using the right tool?" He says, "I don't feel like going out to the truck." You see, tricks are things we do, because we need to get something done fast, and we're going to do it with the wrong materials and the wrong tools, because we don't really feel like going out to the truck. Sometimes by the time you pack up your stuff, you go out to the truck, and grab the right thing, and come back, he had the problem already solved. It doesn't make sense, that's what tricks are.
What we found in our research when we studies this, was that the best teams didn't have a methodology. Or if they had one, they completely ignored it. At one large company we went to, they had an official corporate methodology for design, and they had a vice president, who as far as I could tell, his job was to secretly go to each design room and give them a waiver on following the methodology. "Don't tell any of the other groups, but you guys, because this is a really important project, you don't have to follow the methodology." He just said that to every group.
They didn't follow methodologies, but the struggling organizations, those organizations that really struggle to get stuff out, they all had methodologies, and every time they struggled, their reaction was, we need a better methodology. They were so focused on their methodologies. Instead, the teams that were best at getting stuff out, they focused on techniques and tricks. It was all about building u the toolbox. It was all about trying to get every member of the team to be able to be in a situation and say, "Hey, you know what would help here? This thing, let's just get just do this right now, get it done, and then see where we are from now." They focused on constantly adding new, refining the skills, and like making a roux, practicing with the techniques and tricks. The struggling teams, they didn't have tricks and techniques, they didn't have [inaudible 00:35:14] whatever the methodology provided, and they didn't practice it, they only did it.
This methodological approach is like saying, when I'm cooking, in step 3 I always make a roux even if I'm not going to use it the rest of the recipe. We need the roux as a deliverable for step 3, we always need a roux. How do we claim that preparation is not if we don't have a roux. I know we might not need it later, "Good, you never know when the requirements are going to change and suddenly, we're going to need a roux." That's their thinking, and it doesn't work. In the pontification mode that I'm in, I thought, maybe it's time to get rid of this notion of user centered design. Maybe we should stop thinking in this term. Now, I know there are people that are going to be saying, "Oh my god I just spent the last 5 years training [inaudible 00:36:17]."
What about all the products that are easy to use? Certainly products today in many ways are much easier to use than that display writer was. We've seen great progress, things are better, how did we get here if we didn't think about users in the process, how do we do that? My suggestion is that maybe it's not the process, maybe it's not the methodology, maybe that's not it. Maybe we need to have something like user centered design not for our own purposes, but for other people. For somebody else, and this comes to the story of Stone Soup.
The story of Stone Soup is an old folktale, goes back hundreds of years. It's about a traveler, who was traveling from village to village, and in those days, the tradition was, because there weren't restaurants per se, when you came to a village, you would knock on doors, and you would ask people if they had a place to stay, if you could stay the night, and maybe a little bit of food they could spare. The tradition was, it was good karma, to give people what they have. This particular traveler, on this particular day, in this particular village, started knocking on doors only to find out that the village had been suffering a massive drought and a massive famine, and they were barely able to feed themselves, they just felt they had nothing to give travelers. Everywhere he knocked he got the same answer which was, "We don't have enough for ourselves, so we can't share I'm sorry." There was no other village for hundreds of miles so he was not going to get anything unless he did something drastic.
He decided to make camp in the center of the village, he sees somebody and he says, "I'd like to make some soup, I have the materials here to make some of it, I just need one thing, I just need a pot, could yo get me a pot?" So the person goes off and says, "Yeah I can get a pot." He grabs him a pot, and he's curious because he didn't see anything with him. He brings the pot over and the traveler takes out of his bag, he takes a stone, he takes the stone and puts it in the bottom of the pot. With a stick he starts stirring the stone on the bottom of the pot.
The guy says, "What are you doing?" "I'm making stone soup." So the villager says, "What's that taste like?" Now other villagers are coming around and they're curious because things like this don't happen in the village very often, nothing happens in the village very often. The crowd is gathering, "What's it going to taste like?" The traveler says, "Well you know, it would be better if I had some water." "We've got a little water in the well, I can fill this up, I'd be interested in seeing what's going to happen. They go and get some water and pour it in the kettle, someone lights a fire and they start cooking this thing. "Just water and stone seems weird, it would have better flavor if we had some carrots." He says, "I've got a couple carrots in my parents [inaudible 00:39:34], so he grabs some carrots." The traveler says, "You know meat is also a good flavor." So he goes and grabs meat, potatoes, next thing you know the villagers are all bringing all this stuff and putting it in the thing and cooking up. They've got this big pot of soup, he serves it to everyone.
Everybody enjoys the soup, as a final gift to these people for making this wonderful meal, everybody's really happy, he decides he's going to leave them the stone. So that when they fall under hard times again they can make their own stone soup, and he gives them the stone. That's the story of stone soup. The recipe for making stone soup is fairly well documented, you start with a large very clean stone, and you go through the process. Here's the deal, and this is the really important thing when people talk about the story. The traveler does not believe that that stone makes soup, okay? He does not walk around giving papers on new methods to get soup out of stones.
The stone is an important catalyst to get the community to work together. Having been in this field, and Don usability work and user experience work now, 30 years with literally hundreds of teams I can tell you that the usability test and the [inaudible 00:41:10] has nothing to do with creating a good design. You can create incredibly crappy designs with usability tests and [inaudible 00:41:18]. We've proven that time and time again. Instead, there is something to the people part of this, someone who can facilitate a process of getting people to start talking about what the tasks are and that people might do, and how the people that use this are different from us, and all those things, that's the valuable piece, and if you need the stone to make the soup just to get people to focus just because they're too much caught up in their own thinking, then you use the stone. But when all the travelers get together at their own annual conference, they don't talk about who's stone is better.
It's really quite [inaudible 00:42:08]. If the stone is not ... If the methods are not about having the user in the center what is the goal, the goal is in fact to inform the design process. That's that we're trying to do. We're trying to get design to know something about what's going on. To give you an example about what I mean by this, I'm going to talk about ... A design problem is usually that a design doesn't handle very well. It's a huge problem and it has huge financial implications.
I'm going to talk about a particular big box retailer, I can't tell you who they are, because then I'd have to kill you, but they are a large big box retailer. They have a million people every day, that's close to 400,000,000 visitors every year, come to their website, to buy stuff. They do buy stuff, they buy almost billion and a half dollars worth of stuff every year. There's a lot of money that's being transpired through the design of this website. It's a really important design element. What's really amazing is the conversion rate for this, is 1.6%. Now 1.6% is really low, it's average for a conversion rate, which is funny, but it basically means that 98.4% of the people coming are not purchasing. So only 1.6% ... We get so wrapped up ion these numbers and ... You go to in-conference people and say, "What is your goal for [inaudible 00:43:42]?" "Well we're at 1.6% we'd like to be at 1.7%." What about the other 98% that is failing, not buying things from you. Why do people go to those sites if not to buy something from you. They don't go on a vacation at these websites, what are they doing?
I want to show you something that I made, I've been doing some knitting ... I'm Jared, would you mind helping me for a second? I just want you to stand here and pull this, can you just pull this up? Chris would you mind helping me? I want you to hold this piece up, just hold this up make sure it doesn't come off like that. Can you hold it up high so everybody can see it? This is a piece of string, that represents the million people who come to the website. Every yellow piece of the strip, represents about 15,000 people. This string is actually 67 and half feet long. The reason that it's 67 and half feet long, is because it allows me to demonstrate something. From here, 66 and a half feet. These are the people who come to the site, each one of these is 15,000. Chris quickly, how many customers is that? It's 0, from here, to where [inaudible 00:45:22] is, there isn't a single purchase. All the purchases, 1.2 billion dollars a year, happen in this last foot.
That's the really interesting part about this, but here's the big deal. There's a little 3 inch red piece here, and this 3 inch red piece represents 80% of the revenue. 80% of that 1.2 billion dollars comes from that 3 inches. That's 0.32% of the total length of the string. Is 80% of the revenue that this string represents. What's really quite amazing is that when we look at a daily basis, we get 16,000 customers every day, and 3200 of them represent that 3 inch piece of string. That 3200 is the bulk of the company, each day they produce 2.6 million dollars for the organization. If I could get out of that 3 inches, all I have to focus on is that 3 inches, if I can get out of them another 10%. I'm talking about 100 million dollars in revenue by just focusing on that small group. This is where user centered design fails us, because we have no idea how to find just that. We designed for the entire string which is, in fact, a waste of energy. We can't just design for that 3 inch piece, even though that's where all the money comes from. Okay guys could you please put it down on the ground? Thank you very much. Thank you, round of applause for [inaudible 00:47:28].
That's a real problem here, now ... Part of the problem is we've got this institution issue. There's an old rule of thumb which says, what gets measured, gets done. There's a corollary to that. Which is, what gets rewarded, gets done well. We don't reward in our organization, everybody who's involved in the user experience, on creating great user experiences, we reward them on getting things [shifted 00:48:08], and using certain technologies, and demonstrating tremendous [inaudible 00:48:17]. We don't reward them on doing this stuff ... So we need measures, we need better measures for putting this together, for getting this done well.
To do that, to demonstrate some of the measures that we've been working on in our lab, I wanted ... Another little experiment, just going to take a little preps on how to explain how this would work. We're going to do some experiments in measuring branding picture. [Brand 00:48:49] picture, it turns out, is huge. That little 3 inches, what's amazing about those folks is they are incredibly loyal to the brand. They shop a minimum of 6 times every year on that particular big box retailer site. If we can get them to come just one more time, that's huge. They come to this over all the other choices, they come to this particular retailer because, they love this retailer.
Understanding what that means ... If we can put measures together to do this, of course people have been trying to measure brand loyalty for a long time. The traditional way we do this is by measuring loyalty. Loyalty can be measured in terms of these would somebody recommend your brand to somebody else, things like that.
Turns out there's more to brands, and there's more things you do and we've been working with the folks at the [gala 00:49:48] organization put together, called customer engagement 11, the CE 11. They go farther than loyalty, they start by looking at your confidence and confidence is things like ... is this brand something that I can trust, or do they always do what they promise. Sometimes people define brands on the promise, so this sort of [inaudible 00:50:08]. They go even further than that, they talk about the entirety of the brand, will the brand always treat me fairly, or that if I have a problem I can count a fair and satisfactory solution. Pride becomes a piece that we look at. We look at, am I so proud to be a customer of the brand that I would wear a jacket, or get a tattoo like a Harley customer, and I'll actually tattoo the brand to my ...
Finally, we look at passion, and passion people are so firm about the brand that they will [inaudible 00:50:46]. We can measure into these things, today I have done just that. If you are one of the people who got the little survey, what I want you to do at this point, is if you've totaled up your columns on each page, I want you to subtract the column that had the agree ... It should be agree minus disagree, only 40 of you know what the hell I'm talking about right now. For each page to agree minus disagree, this means if you had more disagrees you're going to total a negative number. So it will be a number between 11 and -11. When you've done that, I'd like you to come up to the front of the room where I've carefully created a scale, and we're going to do a little experiment. Come on, for all 4. We're going to start ...
So the first question I asked you was about Starbucks. I'd like you to get next to the number that you put for Starbucks, on this scale, -11 +11, we're going to create a human pie chart. If you strongly disagree you'll be on the negative side, if you strongly agree you're positive. So in general what I'm seeing here as our data comes in ... That's the positive side, this is positive that's negative. We got that? This is basic math kid. Okay, so what I'm seeing here is a real bell distribution, look we got a couple people here, and it all bunches up around -2, and then it pales off over here. So we have created a nice bell distribution. I'm actually pretty surprised, when I do this in Boston I do a Dunkin Donuts and it's all skewed. This is actually interesting, it's ... Jacob, can you tell me, say really loud, why do you love Starbucks?
Jacob: I just like the taste.
Jared Spool: So it's just the taste? Let's go down here, let's find somebody down here. Why is it that you don't like Starbucks?
Speaker 4: I'm not sure if it's organic.
Jared Spool: You're not sure if it's organic, Anders, why ...
Speaker 5: It's infesting Manhattan.
Jared Spool: It's infesting Manhattan, interesting. Okay, so it's not necessarily organic but even if it wasn't ... You wouldn't let an organic infestation of Manhattan? Okay, so the second one I asked them for was McDonald's. Can you find your number for McDonald's. Let's see where that distribution is. Okay, well there is nobody above a 5. This is a flatter distribution if you look, it's not a bell distribution it's a flatter distribution that is heavily skewed to the negative here. So what did McDonald's do to you guys?
Speaker 6: It infested the world.
Jared Spool: It infested the world, okay.
Speaker 7: I just don't go to McDonald's.
Jared Spool: Just don't go to McDonald's? This is not a "Don't go to McDonald's." This is "I hate McDonald's, I want to see it taken from the planet." Don't go to McDonald's is over here. This is interesting, the next one I asked is Apple. Find your position for Apple computer. Look at how many people are in the rear left. We have actually somewhat of a bipolar distribution here. When I do this with my students it's almost always a bipolar distribution. Okay last one, Microsoft. What is wrong with you people? You guys work for Microsoft? No, okay. This is available in Second Life right now. Look at this distribution. Again, this is very heavy on towards the negative, very heavy on the neutral. Look at how easy it is to measure brand. Everybody give a round of applause for these folks.
We did this, while we were having people [shop 00:56:26]. These are our [compelled 00:56:27] shopping [inaudible 00:56:28]. We actually have these e-commerece tests where we find people who are ready to buy a product, we bring them the sites that have those products, we give them the cash to buy the product, yes you can sign up for these studies on our website, you get to keep the product. We do this and we tested different electronics brands. For this type of thing. We looked at 5 different brands, Amazon, Circuit City, Dell, HP, Wal-Mart, actually the study was 13 but I'm only talking about 4 here because that's all I have room for on the screen. Would you mind pushing those easels to the side there ... When I did this with Amazon ... The neat thing about this instrument that we've created, is we can measure brand engagement in different points in time, in the use of the site. So as people are shopping we can consistently get their feedback.
So at Amazon, when they started, before they had actually done the shopping, they just came into the lab, the average score for 72 people in our study, was 6.2. When they were done and actually purchased their product, or decided they weren't going to purchase from Amazon, it dropped to 5.5. So we saw a decent sized drop in the Amazon score.
Circuit City also started 4.5, but we saw a much smaller drop in their score. They were still disappointed but not as disappointed as Amazon was for whatever reason. The Experience didn't change the brand.
Dell went from 3, down to 1.4. Now keep in mind, these are people who we'd given $1200 to to buy a laptop. They were buying a laptop, in essence for free for participating, and that's what they felt about Dell.
At HP, it went from 1.4, down to -1. Same deal as Dell, but people did not like the HP purchasing experience. Guess where Wal-Mart buds? The thing about Wal-Mart is, it started really low, people came in like, "Oh my god I can't believe you're making me buy this laptop at Wal-Mart." When they were done, it went up to 1.1. Not a huge jump, it is a huge jump from .5 to 1.1, much bigger than the drop from 6.2 to 5.5, from a percentage perspective. It is not a big absolute raise, but it's up. You can raise people's expectations. The Wal-Mart way of raising expectations is not through great design, but just for setting incredibly low expectations. That's a method, the key to success to reduce expectations is a tried and true business practice.
What's really interesting about this ... The second measure came from when they were done with the site, but hadn't received their buy yet. We called them a few weeks later and had them fill out the same paperwork and tell us what they thought of the brand after they received their product and dealt with whatever they had to deal with. What we found was that every brand that's listed here dropped in that experience, including Wal-Mart which went down to 0.1. They ruined whatever good experience they had but actually doing crappy in the filming.
Experience design, which is the whole process, not just a piece of it, and that's really important. This is just one measure that we've been playing with, I could go on about this and that will give me stuff to talk about next year, but we have to be careful because there's a lot of voodoo techniques out there. There's a lot of things that look like we're measuring important things, but we're not. My first [candidate 01:00:26] for this, is eye tracking. Eye tracking is really pretty, it produces the best deliverables, because they have all these different colors in them and they look really scientific.
This I stole off of ... I don't know if you've heard about the guy, Jacob [inaudible 01:00:49]. He has all these charts on his site, this is one of the ones that were 99% good so I took the chart. He did this eye tracking stuff, they're big into eye tracking these days, they've proven that guys will look at people's crotches and stuff like that. Which is an important finding if you're talking about a website with crotches on it I guess. He produced this chart, which is a [e-commerce 01:01:25] page, it's real hard to see with the contrast here. What it says is, in this shopping cart, users didn't look much at the cross selling offers as this stuff down at the bottom. Which is a common finding, basically what he's saying is eye tracking proves that people don't look at the things we already knew they didn't look at it, because for one thing, nobody buys the goddamn cross selling products. We know they don't look at it, they don't sell. The thing is we needed a $10,000 piece of equipment and a $30,000 consultant to actually confirm this.
That's the problem with this methodology, it actually doesn't tell you anything, in fact, when I was in the CHI conference, there were all these vendors in te exhibit space, had all these great visualization tools for eye tracking. This is what really [inaudible 01:02:15] from my square, as you move through the site, the time people spend on the pages, the little circles start to get bigger and bigger and bigger, and the little diamonds represent where a different user is looking at a time, it's really funky to watch. I had no idea what this thing is talking about but I could sit there and I could watch it all day long, it's so much fun. It's my twitter bid, which is an application that puts out messages and twitters all over the world. There is no real reason to spend ...
That's what eye tracking is for, it's to get you excited about actually doing the real work. Maybe this is the stone in the soup, in fact, at Google, the head of User Research there told me that they have 2 labs, 1 has eye tracking equipment and 1 doesn't. From her perspective eye tracking is great because the developers will shut up and go along with the eye tracking. She cant get to the company lab she schedules a test for the lab without eye tracking, the developers don't show, but if she puts it in the eye tracking lab they do. Apparently developers, when they're not working on [inaudible 01:03:23], like following little blue dots. In fact [inaudible 01:03:29] is camped out a lot and won't leave. To her this the most valuable thing about eye tracking because it gets the developers in the lab. If that's what it's for then we need to be honest with ourselves, if that's what eye tracking is about then yeah, let's do it.
But that's now how I hear it's [sold 01:03:47]. Another one of this is site analytics. Site analytics produces these incredibly charts for which we both know what they need. Here, there's a big dot, is that because the user is interested in the content, or is it because they're lost. Either one will produce more time on the site, which one is it? We can't tell from that data point. Here, are they bored, or was it really hard to get information they got what they needed then they got out of there. We can't tell, so the analytics don't actually tell us anything but man, you can change the scale and you can move the thing, you can go back and forth, it's fun. It's like tetris for real work. All it's missing is a high score file. Then everybody will be looking at the analytics.
That's the question, is this bad, or is this safe? Which is it, this is really no different than the TSA stuff, it's the same sort of fear. I've got 6 minutes left, I'll give you a [inaudible 01:05:00]. Our research so far, has taught us that there are 3 core attributes that a team needs to have to be in that group that makes them really successful. Those 3 attributes are a solid vision, a very strong research feedback capability, and the right culture for doing this type of work. There are actually different types of cultures, but not all of them succeed, there are several that do.
We can boil these 3 attributes down to some key questions, and we now use these key questions to basically tell us where the organization is on our scale of success to failure, incredibly predict. The correlation of these 3 questions is really strong. For vision the question is, can you go up to anyone on the team and ask them, "What will it be like to use the experience of this design 5 years from now," and get a coherent answer, not just a coherent answer, but one that matches everybody else's answer on the team.
If you can do that, then you've got this vision thing handled. Think of a vision as a stake in the sand, on the horizon, but you can't get to today, it will take 5 years to get there, but everybody can clearly see it. That means that every step you take, you can tell whether you're moving towards it or not. As a group even though you're not really talking about it out loud, you can each tell if you're moving towards that flag, or away from that flag. That's what it turns out to be, that's what vision is about. Sometimes you have to take little steps backwards to be a [inaudible 01:06:52] when you're crossing a creek. Sometimes it snows and the creek don't go right where you want but that's okay, because you can see the flag and know you'll get there eventually. That's the importance of vision.
Feedback, in the last 6 weeks, 6 weeks is a really important number, in the last 6 weeks, has anyone one your team spent more than 2 hours, another important number, actually watching the people who will be your users, do the things that they will do when you [inaudible 01:07:21]. I suggest asking at any point in time in your schedule, and you should be able to give me a solid yes. In fact, I should be able to ask anyone in the organization, and that person will be able to give me a solid yes.
Those organizations that do that ... Now what's really interesting is while [inaudible 01:07:39] to the world, are not doing usability testing, they're spending a lot of time watching people use their product. When Apple was developing the iPhone, they spent more than 2 hours every 6 weeks watching people use those phones. It was very clear it doesn't have to be formalized methodology doing this, but, spending the time watching, not talking, not [inaudible 01:08:09], not explaining, watching and listening is key. Having an organization that is set up to do that constantly, is key.
That brings us to culture. Culture is in the last 6 weeks, has someone in your team been rewarded for really fucking up. Some organizations that is rewarded consistently but not in the same way. This is referent of major design failures. Few years back I attended a presentation by Scott Cook, the founder of Intuit, who talked about how the CEO, he regularly holds failure parties, where they take people who created big design problems and reward them, giving them huge gifts and having a celebration. Part of the purpose of that celebration, is to talk about why this was such an important thing to learn about a customer, and why now that they've learned it, they can be such a better organization.
You have to have an organizational culture that thinks that way. If you're in a culture where any sort of failure is seen as a bad idea, and we must do everything we can to prevent it upfront, we can't possibly learn. As the old saying goes, good judgement comes from experience, and and experience comes from bad judgement. We need to able to have that [inaudible 01:09:42]. That's what I wanted to talk to you about today, I'm thinking at this point, it is definitely time to start retiring, at least amongst ourselves, this notion of user centered design. No one can define what it is, no one can explain it, no one knows how it works, it doesn't really exist. If we want to use it as the stone let's just agree that's what we're going to do and we can at least stop the damn sessions about it and conference. It doesn't really exist ...
Instead let's focus on inform design, what information does the design team need to make great decisions and what techniques and tricks do we have in our toolbox that let us get there. As you're going through the sessions through the next 3 days, think in terms of techniques and tricks, not methodology and dogma. Think in terms of, how can I take a little bit of this and how can do a little bit of that, and where could I use this and think in a way the presenter doesn't even think that this is a great way to do this. How am I going to be able to put that forward. Focus on these 3 core [UX 01:10:48] attributes.
Do you have a vision, if you don't have a vision that's the first thing you need to be working on. You should be regularly talking about the vision, making sure everybody knows what's going on. You can always talk ... It's very hard to get information out of Apple. I've talked to a lot of Apple employees as soon as you start to ask them any questions they tell me right off, there are ninjas stationed all over with little darts. They're not allowed to talk about anything. Family members regularly come in, my husband works for Apple [inaudible 01:11:24], not only do they have a vision for their products, but it goes out [crosstalk 01:11:33], and they all know what it is. They're there.
Do you have feedback mechanisms, are you getting information, you can do it in the most guerrilla, shoe-string form. It doesn't have to be excessive research. You don't need the fancy lab with the one way mirror, by the way it's a one way mirror not a two way mirror that's a window. You don't need the fancy lab, you don't need that stuff, it's an awful waste of furniture in class, we have been testing in the lab for a hundred years, you don't have to do it that way. Instead, you just need to sit and watch people. You can do it at a Starbucks, people here don't like ...
Finally, do you have the right culture, there's different types of culture. At Apple, they have a culture of, if I don't get this design right, Steve will have my ass, that's the culture, what would Steve do, that's the culture. Every employee is expecting to have to talk to Steve at some point, and explain their design rationale, and they better damn well be prepared for it. So everybody focuses on that mentality.
I've been asking people, you can get them to talk about certain things. I have a question, in fact, whenever you meet an Apple employee ask them this question, you'll get an answer from him. In Apple, the mecca of good design, what is the software, what is it like to use the software that you use the software that you use for internal [expense 01:13:05] recording. Is it an incredible piece of expensive recording software? Turns out everyone's free to talk about this, it sucks. It's sucks really bad, no one likes using it. Here we are at this place where good design is the hallmark of their success, and their internal pool, sucks. I ask each person I say, "Why do you think that is?" They all have exactly the same answer, "Steve doesn't do his own expenses." The [inaudible 01:13:41] recording software, it rocks. It's really slick, that's what steam does.
That's the culture, but that's not the only culture, you go over to Netflix, a company that's now 3 times the size Blockbuster because they built a cool website. Netflix, doesn't have that, Reed Hastings doesn't run Netflix with the fear of god behind it. They have a very completely different culture about experimenting and trying things out, and putting out a new release of their website every 2 weeks and seeing what happens. That's how they do it. Lots of different cultures were, we're trying to catalog these and understand them, but you have to find what works for you.
That's what I'm here to talk about, if for some reason you found this interesting, we publish a lot of the things we find in our newsletter. If you're not signed up you can give me your email address I'll sign you up, or you can sign up on our website. If you want a copy of these slides I will send you a copy just give me something with your email address on it, I'll probably just take the mp3 file they they're making and put it up on slide-share. I'll send you an email link to let you know when that's available, so you'll have the full audio and the slides. We also have virtual seminars, [inaudible 01:15:00] a seminar in mid May, I don't remember the exact date, I think it's the 14th, don't quote me on that, that talks about brand stuff we just talked about. We have a ton of variety of things we have our User Interface Conference with [Jet 01:15:14], it's okay if you [inaudible 01:15:16] for the User Interface Conference for the competition. Cool.
Everybody who registers and pays, unfortunately the free person won't get this, you're in high school. Everybody who registers and pays for the early registration tonight we're giving away a flip video, it's the really cool little video thing that you ... The newest cool thing, everybody gets that. We have our blog with contacts and stuff like that, we have our website uie.com which I'm desperately hoping is usable, thank you very much for a [crosstalk 01:15:52].