In a November 13, 2007 press release headlined, “Direct Comparison of iPhone and Hard-Key QWERTY Phone Owners Indicates Higher Text Entry Error Rate for iPhones,” User Centric, Inc., a Chicago-based usability consultancy, states that they have finished a third and final study examining the user experience of Apple’s iPhone.
Our study involved data from 60 participants who were asked to enter specific text messages and complete several mobile device tasks. Twenty of these participants were iPhone owners who owned their phones for at least one month. Twenty more participants were owners of traditional hard-key QWERTY phones and another twenty were owners of numeric phones who used the “multi-tap” method of text entry.
MacDailyNews Take: Everything you are about to read from this point on from User Centric’s press release (in italics) is virtually meaningless and applies only to a miniscule sample of 20 people who have owned an iPhone for at least a month plus 40 other non-iPhone users. The likelihood of the sample size being too small to be useful to anyone outside this small group of 60 people is extremely high. We could pick another random 20 iPhone users and get completely different results. Were these 20 iPhone owners selected because they were heavy users of text messaging or did they get their iPhones primarily so they could browse the real Web or perhaps so they could have a multi-touch iPod or maybe they use email over SMS texting? How many, if any, of these iPhone users even use their iPhone regularly or at all for texting? What were the ages of these 20 iPhone owners? Was it a group of 35-year-old soccer moms who don’t know SMS from PMS or were they 16-year-olds who text in their sleep? By the way, how long were their fingernails? The questions are endless. Without such salient details being disclosed, the tiny sample size of 20 people becomes even more of an issue. This “study” is fatally flawed.
Participants were brought in for 75 minute one-on-one usability sessions with a moderator. Each participant entered six fixed-length text messages on their own phone. Non-iPhone owners also did six messages each on the iPhone and a phone of the “opposite” type. The opposite phone for numeric phone owners was a Blackberry and for hard-key QWERTY phone owners it was a numeric Samsung E300 phone.
MacDailyNews Take: Oh, so iPhone users got to enter 6 messages on their iPhones, but not the other phones? However, the other 40 people who had never touched an iPhone got to enter 6 messages on the iPhones? Why? To inflate the number of “mistakes” made on the iPhones?
When compared to hard-key QWERTY phone owners using their personal phones, iPhone owners’ rate of text entry on the iPhone was equally rapid. However, iPhone owners made more errors during text entry and also left significantly more errors in the completed messages.
MacDailyNews Take: The full press release details the meaningless stats that this tiny, undefined sample of just 20 iPhone users generated.
Furthermore, when iPhone owners were asked to perform a text correction task during their sessions, 21% of iPhone owners were not aware of the magnifying glass correction feature although they had owned their iPhone for one month. Participants who did know about the feature clearly loved it, and participants who were new to it indicated that it would be useful in the future.
MacDailyNews Take: First of all, how do you get “21%” out of a sample of 20 iPhone owners? 4.2 of the iPhone owners didn’t know about the magnifying glass feature? What happened to the other 0.8 of that individual iPhone user? Maybe he thought he knew about it, but forgot? So, the sample size is too small and too undefined and now it’s flawed to boot: Tell all of the participants about the tools available or your study’s results are junk. The fact is that the magnifying glass tool is available and all users should have been informed and allowed to use it. This would have allowed the survey to gather meaningful results if, of course, they had used a reasonable sample size, which they didn’t. Did they also neglect to tell 21% of the hard-key keyboard users where the arrow keys were or how to use them?
Participants who had previously not used either a hard-key QWERTY phone or an iPhone were significantly faster at entering text messages on the hard-key QWERTY test phone than on the iPhone. These participants also made significantly fewer errors on the hard-key QWERTY than on the iPhone.
MacDailyNews Take: So, had these participants used a hard-key QWERTY keyboard on something else like, say, a personal computer? Of course they had, and likely for many, many years. Certainly most people have vastly more experience with hard-key keyboards vs. users of iPhone keyboards with “at least a month” of experience. That experience gap would be accounted for in a real study. It is ignored here. All of the participants are naturally inclined to be more familiar with a hard QWERTY keyboard whether it be on a phone or not. It’s only logical. But, the last thing those who conducted this “study” want you to employ is logic; either that or they’re just incompetent.
Numeric phone owners made an average of 5.4 errors/message on the iPhone, 1.2 errors/message on the QWERTY test phone, and 1.4 errors/message on their own phone. “Not only was their performance better,” says Jen Allen, User Experience Specialist for User Centric, “their rankings and ratings of the phones indicated that they preferred a hard-key QWERTY phone for texting.”
MacDailyNews Take: Oh, so those familiar with QWERTY keyboards and wholly unfamiliar with iPhone’s keyboard performed better on a QWERTY hard-key keyboard and ranked it higher? Stunning finding there, Jen. One would also guess that there’s an 1879 “study” in a dusty file cabinet somewhere that found people preferred candles to electric light bulbs for reading. No doubt commissioned by the wax industry.
User Centric also conducted a “detailed analysis of text entry patterns” on 20 random people which came up with a bunch more meaningless numbers which are used to either obfuscate the appalling ridiculousness of the study’s useless sample size and methodology or, again, they’re just so incompetent that they think the numbers have meaning.
Compared to hard-key QWERTY devices, the iPhone may fall short for consumers who use on their mobile device heavily for email and text messaging.
MacDailyNews Take: Or iPhone may not fall short. The iPhone may be a heavy email and text messaging user’s godsend. We simply don’t know either way from this study. Among many other faults, the sample size is too small and too undefined and they didn’t let the iPhone users touch the other phones.
The iPhone was clearly associated with higher text entry error rates than a hard-key QWERTY phone.
MacDailyNews Take: Only in this particular “study” with its tiny sample size and flawed methodology: Letting non-iPhone users contribute mistakes to iPhones while keeping iPhone users off the other phones, thereby logically inflating the non-iPhones’ accuracy. Hard-key users, regardless of type of phone keyboard would naturally be more familiar with hard-key keyboards and therefore make fewer mistakes in use.
“The iPhone is a great switch from a numeric phone. But if you’re switching from a hard-key QWERTY phone, try the iPhone in the store first,” recommended Gavin Lew, User Centric’s Managing Director.
MacDailyNews Take: More garbage, which is a fitting end. Trying an iPhone in a store sounds like a great idea, but you need to use it for at least a few days to learn how it works and get used to it (and it to you). If you base your decision on how you typed on an iPhone in a store for a few minutes, most likely your conclusion will be exactly the wrong one – which conveniently seems to support the findings of this “study.” It’s no wonder Lew wants buyers to base their opinion of iPhone’s keyboard on laughably insufficient testing, as that’s exactly what his firm’s “study” does.
We encourage readers to read the full press release while employing logic and paying close attention to which users were allowed to use which device(s), how few people were involved, and what exactly was tested, here.
We also encourage readers to note any media outlets that dutifully report the “findings” of this “study” as fact.
MacDailyNews Note: On their website, User Centric lists their “clients over the last two quarters” which include: Verizon Wireless, Microsoft, LG, and Motorola.
owned
Actually this is one of the few functions of the iPhone that are an issue for me. Being a fanatic so to speak I love the iPhone of course but typing while driving sure can be a problem. I know I shouldn’t!
Also please come out with a copy and paste function!!!!!!
Oh yeah and a function to add a contact from the recent calls list that automatically carries the phone number over!
My question is who is paying for this study? I have my suspicions, and I am sure I am not alone in them.
Direct comparisons of eating iPhone glass and eating ice cream show clearly that the iPhone sucks.
For our next direct comparison, we will compare iPhone’s smooth surface to the Himalayas and Jessica Simpson’s face before Proactiv Solution.
How about a direct comparison between other all touch screen surfaces? Ugh.
The premise of this study is obvious to test the iPhone, not the other phones.
I could call it a real study if EACH user typed that same message on a different phone each day during the study (to prevent thumbs from getting tired).
THEN if they said the iPhone produced significant errors, I could believe it. As it is, this study is just plain BAKA! (Japanese: stupid)
I’m only a sample of one, but I love the touch keyboard – I furiously type without regard for accuracy – the predictive correction works wonderfully for me except for proper names (duh), where I have to slow down and eyeball the word. The newest feature of double spacing adding a period makes 99% of texting very easy too
Note also that iPhone users had their phones for “at least one month” (most may have been relatively new at it) while the hard-key phone users had their phones for an unstated period of time (very likely a much longer time.) So they may well be testing people with a ton of experience on hard-key phones against people who never used a phone keyboard of any kind before they bought an iPhone. We just can’t tell from the information provided. As a former ad agency guy who has witnessed decades of bogus testing masquerading as legitimate research, I can tell you — I smell something extremely fishy here.
This “study” sounds like the news from Reuters… total CROCK!
It really suck to read and article when Mac Daily News is stuffing their two cents worth in after every sentence. By the way, I didn’t read your take because we all know you are way to far up apple’s ass to accept that any apple product may have a shortcoming.
Let us read the article first then make your arguments.
“Oh yeah and a function to add a contact from the recent calls list that automatically carries the phone number over!”
You can do this on the iPhone. In your Recent Calls list, press the blue arrow beside the phone number, then press the ‘Create New Contact’ or ‘Add to Existing Contact’ as suits your fancy. Either way, the phone number automatically carries over, as you would expect.
One would also guess that there’s an 1879 “study” in a dusty file cabinet somewhere that found people preferred candles to electric light bulbs for reading. No doubt commissioned by the wax industry.
MDN = Brilliant bastards!
Typing on the iPhone is an absolute nightmare compared to every other PDA I have ever used. And I’ve used several (Treo 600, 650, 700, blackberries since the pager-lookin-ones, Visor).
Miskeys are frequent, the “predictive text” is wrong at least 50% of the time, typing in a moving vehicle (back seat of a taxi, not while driving) makes accuracy 3-4 times worse, can’t text while walking or talking to someone else, and no matter how carefully I try to proof, my messages ALWAYS go out with at least one mistake. I’ve recently changed my sig to “Typed on an iPhone. Please excuse typos.”
And while I’m at it, how come I can’t address an SMS to more than one person at a time? That’s just asinine.
Whew. I feel better. Anyone wanna buy a used iPhone?
Troy,
It sounds like your reading skills are being over taxed. Maybe you should just stick to your coloring books. You need to accept your limitations, little boy.
@ Don Jakubczak:
You can create a contact from the recent calls list without typing the phone number.
On the list of recent calls: Press the little blue arrow on the right side of the phone number, which will pull up a menu. Press create new contact.
Troy:
Just out of curiosity, if you don’t want MDN’s opinion, why do you read MDN? You could get your news in millions of other places . . .
@Piece of Crap Apologist
Funny your comments about predictive text being wrong 50% of the time–I find it gets things right at least 90% of the time. I’ve had my iPhone from Day 1, and I couldn’t be happier with it. I seriously doubt you’ve ever touched an iPhone, much less used it to type in a moving vehicle, etc.
Let’s see, what percentage of iPhone owners are satisfied with it? Oh, yeah, something like 90%.
Go suck on a beet!
“Numeric phone owners…preferred a hard-key QWERTY phone for texting”
That only suggests that the adjustment from a physical numerica keypad to a phyical (or “buttony”) QWERTY is more comfortable.
The vitural keypad is very different – the learning curve is different because it is not only transfering from numeric to QWERTY, but from buttons to a glass pad only.
“…try the iPhone in the store first,”
Great, except wasn’t the point of one of their earlier “studies” that it took a while to get used to the virtual keypad? How can trying it in a store be a useful indicator of whether one would like or dislike it?
The real question, neither asked nor answered here, is whether one can become as proficient at typing with a virtual keypad as with a physical one. The answer is very likely yes. People are very good at adapting, esp with something that is not very difficult and with enforced daily practice.
Troy,
While other outlets dutifully report the finding of this “study” as gospel, MacDailyNews obliterates the study and it’s “findings” with exquisite logic.
I’ll take my “news” from MDN over the others any damn day of the week.
MDN has the truth on their side.
MDN may be right about study flaws (did they really count non-iPhone users in iPhone errors but not count iPhone users for other device errors?). Certainly the comment about “try the iPhone in the store first” is an idiot comment from the researches, because someone in the store will obviously not have gained expertise and won’t be inclined to like it on that basis.
Nevertheless, the sample size of 20 is NOT obviously too small a sample. MDN, you are being unfair or you don’t know your statistics. You can’t evaluate the adequacy of their sample size unless you know the variance they found or the p-values in their significance testing. With the large difference in error rates they found, in my experience a sample size of 20 is likely to have been more than adequate.
Don’t criticize aspects where you don’t know what you’re talking about.
Amateur
For a usability study, 20 people is actually a lot of people. The mystery about this story is why was a usability consultancy studying only keyboarding, which I’m not so sure really falls into the category of usability anyway, rather than the overall usability of the iPhone compared to the other phones? I think it’s pretty clear how the iPhone rates in overall usabiity since most iPhone owners can use virtually every function of the phone without reading a manual while studies reportedly show that most functions of traditional cell phones , smart or not, go unused due to the complexity of using them.
Our study involved data from 60 participants who were asked to enter specific text messages
That phraseology immediately rings alarm bells and I did not see anything to diffuse my suspicion. The word that appears to shed light but actually causes confusion is ‘specific’, specific to what I would ask. It could mean that the messages were specific to each kind of phone.
The report further adds to suspicions by the survey’s methodology, The order of phones and text messages were counterbalanced across participants to prevent ordering effects. No, to prevent ordering effects all participants should have followed the same order. The methodology introduced ordering effects.
<i>They also typed two pangrams – a sentence that includes every letter in the English language at least once – and one corpus – a set of characters that represents the exact letter frequencies of the English language. These tasks were included to ensure that participants experienced the various phone keyboards in a thorough manner.<i> These sentences serve to appear to confirm fairness and equality but in fact do no such thing.
Given the obviously flawed nature of the methodology and the doublespeak used in the report it seems suspiciously like the test was designed to fail the selected model. How it could be done whilst fitting the stated methodology:
Give iPhone users more often misspelled words
Give iPhone users longer text messages first and shorter later
Give iPhone users messages where the reach between consecutive letters is on average greater.
Give iPhone users pangrams with more x,y,z etc
&etc;The message I receive from reading the report is not that the survey was flawed but that it was fraud.
This study just has to be bought and paid for by rival handset maker(s) and carrier(s) who do not have iPhone.
Follow the money.
MDN is right. Everything about the study is wrong.
My question is, who funded the study?
I don’t have an iPhone but I do have an iPod touch.
Typing on my iPod touch is the better than typing on any phone I’ve ever owned.