ShopTalk Blog

Intuitively, we all know that flipping a fair coin results in heads 50% of the time. But how many times should we expect to flip a coin before the stats converge on that number? I turn to my esteemed statistician friend, Christoffer Perry, with such questions. He says:

In Statistics, everyone seems to think that 95% is a magical number, so let's pretend like we're statisticians (a dubious proposition, at least for myself) and bow down to the mythical sphinxian ninety-five. Conversely, 5% is a pretty good number in the discipline too, so we'll use the fabled feathered five. That is, for those of you not privy to my inner thoughts and schemes, let's say we want the probability of the sum of heads falling within five percent of the average to equal ninety-five percent.

Thanks to R, the Jason (of Rgonaut fame) of Stats, we soon find that you'd have to toss a fair coin about 1,500 times. That is, there's roughly a ninety-five percent chance that the number of heads will be somewhere between 712 and 788, five percent off 750. That's a lot of coin tossing for not a lot of certainty (though you can be certain that rattling off this sort of probability at a party will get you quickly tossed).

To get even more certain, say, 99% probability, you'd need about 2,700 tosses. To be more certain and get a range of one percent, you're looking at somewhere near 66,000 tosses. But hey, at least you got a lot of free drinks at those parties!

A fair coin is a random variable. Your users are not.

At Startup School 2007, Max Levchin advised the attendees to take a course in statistics and realize that a sample size of 30 is too small to produce statistically significant results. That would be true if you were sampling a random variable. Unfortunately, this was his argument against usability studies, which sample human behavior.

In college, I was fortunate enough to study under the wise tutelage of Dr. James D. Hollan. I spent an entire quarter redesigning a pretty unique piece of video viewing/editing software developed at Stanford, called Diver. The first assignment required us to conduct 16 contextual interviews with actual Diver users. Each interview consisted of at least two members of our six-person team sitting and closely observing a Diver user, in their office, for two to three hours. That's a lot of man hours for the first assignment in the course.

The first several interviews were basically shock and awe. All of our expectations about user behavior were shattered. We never could have predicted how these people were using the software. Breakdowns and problem areas in Diver's interface shot out of the screen at us like Captain Eo's lasers. Possible changes and improvements raced through our minds. We had chosen wisely; this software was ripe for a redesign. However, after about eleven interviews we came to a pretty obvious conclusion: all of the users behaved roughly the same. After learning a ton from the first few interviews, we now felt that we weren't learning anything new at all by conducting more of them. We took this observation to Dr. Hollan. His response was that this pattern is entirely typical. He teaches this course every year to hundreds of students, and they all have the same experience. He asks the students to conduct 16 interviews, but that is always more than is necessary. We were allowed to stop at eleven, and we all breathed a sigh of relief.

The lesson learned: human behavior converges on its expected value far more quickly than a random variable does.

Applying this lesson

Reading words on buttons and menus requires a lot of brain power. The image of the words needs to go from the eye to the brain. Then it needs to go to a different section of the brain for language processing. Finally, another section of your brain decides if those words represent what you're currently searching for and whether to signal the hand to click on them. Icons, on the other hand, can be more easily differentiated and can allow users to short-circuit this process. They can notice the icon in their periphery and click on it without putting much thought into it. That means their thoughts can remain on what they're trying to accomplish instead of how to accomplish it. For this reason, the room tabs in the first version of ShopTalk looked like this:

Every single one of the first 5 users was confused by that little tab on the right. Nobody clicked on it. For me, that was already enough data to warrant fixing it. After just 5 tests I already felt close to that 95% certainty that required 1,500 coin flips. I'm willing to bet that you, kind reader, are also confused by it at this very moment. We humans all think alike. Your users, in this sense, are like a gaggle of Lemmings.

Our task as interface designers is to guide those lemmings to safety. You have to put up the proper signs and road blocks so that they think what you want them to think and do what you want them to do. You also have to guide them with such a soft touch that they don't even realize they're being guided. In this case, the fix was rather simple:

Labeling the tab with text and an icon is often a great combination, if you can afford the screen real estate. First time users can read the words to figure out what the icons mean, and more seasoned users can spot that little green plus sign out of the corner of their eye and click on it without even thinking about it. The result: happy users who keep coming back for more and tell all of their friends to check out ShopTalk.