Don't use the general public as guinea pigs
Published by Steve Litchfield at 11:29 UTC, November 7th 2008
Year after year, handheld and phone companies make the same mistake, releasing buggy and incomplete products to market and then having to back pedal and fight fires for the next 12 months. Back when I was in industry, our company had a motto: "Right First Time". Now, I know phones are complex devices, but there's still got to be a better way. Read on....
One of the ideas behind "Right First Time", as you might guess, is that you ultimately save money and time by making sure that the product you release is as perfect as possible before letting the customer have it. Looking across the phone world, we've seen the original Nokia N95, the new N96, the T-Mobile Android-powered G1 and and a rather large number of Windows Mobile-powered devices, all made available on the High Street before they'd actually been finished.
The attitude of the manufacturers seems to be "Oh well, we must get to market first, it won't matter if it's buggy, we can always fix things later and update customer's firmware". But what about the huge damage to reputation caused by those first 1% of customers being repeatedly let down by flaky software and spreading the word: "Avoid"?
Internally, most of these phones - I'll use the N96 as my example here - must undergo some testing. Nokia probably has test suites that have to be run on each device that gets shipped. And no doubt a handful of real people who play around at weekends and try to spot the bugs they can. But nothing, repeat nothing beats the stress testing that a batch of typical early adopters will put a product through. In the case of the N96, you've only got to load up 20 apps, 1000 songs and a few movies and then start to use the device as a phone all day, every day, to discover instabilities and problems at the rate of about one an hour. Would that be so hard for Nokia to pick up back at base?
In addition to the automated test suites, Nokia would do well to formally go through what they did informally for the release of the N95 8GB - i.e. recruit a number of real world power users in a 'closed beta' (ok, ok, so I was one of them) and tell us to 'break it'. The automated test suites will catch some bugs, but certainly not all. It takes the randomness and unpredictability of a real user wanting to do real things with a phone, if you want to start catching the bugs that will also catch out Joe Public when he buys in the High Street.
Whichever way manufacturers choose to test, it simply isn't acceptable to have them relying on the aforementioned (clueless) Joe to be the guinea pig for a device that's known to be unstable. I'm a geek, you're a geek (probably), and we can handle various errors and crashes on early firmware. We know how to work round problems, we know how to back up, how to reset, how to check for and then update firmware. Joe doesn't.
Ideally, companies like Nokia should do more real user tests in-house. Failing that, go down the closed beta route. But please don't actually make a phone available to the mass market until it's stable and reliable. The N96 is, unfortunately, a classic example of how not to do it. Even the latest v11 firmware is horribly flaky and yet most N96 users in the UK acquired it as a 'finished product' from their network or High Street store and are simply left with the impression that the N96 is 'disappointing'.
Whereas the average power user on AAS would probably call it 'disappointing, but with an awful lot of potential once Nokia have taken the extra 3 months to actually finish its software'.
In the meantime, if Nokia, Sony Ericsson, Samsung, etc. would like to recruit a number of real world phone stress testers, they know where to look.....
Steve Litchfield, 7 Nov 2008
Categories: Hardware, Editorial Thoughts
Platforms: General
News Discussion
Sorry to be off topic .
But where exactly is High Street? :-)
I do understand the connotation, but is it a real place that you are referring to?
I am put off buying another until firmware comes out to fix it (assuming it does).
This is something that really should have been spotted before release.
|
Originally Posted by bartmanekul
Exactly. The N85 has very bad camera noise on almost any low light situation.
|
Though this takes me to my other point. It's all very well saying that testing saves money but you have to make money first in order to save it. For the networks, making money is all about getting people signed up to contracts and, for that, the latest phone is one of their biggest attractions. If Nokia (or whomever) takes too long testing a device before it is released to the networks it could be old hat.
Just look at SonyEricsson and the Xperia X1. They have spent forever ironing out the performance issues and bugs. It has finally been released to general appathy as the features that were revolutionary when announced 9 months ago are now available on a dozen other devices.
This is a problem O2 also suffer. Due to their testing standards, other networks often get a 2 month head start on them with new devices and so acheive better sales. So, perhaps it isn't the manufacturers fault or the networks fault. Maybe us consumers are just too willing to forgive previous trespasses when we see the latest shiny gadget. If we really DID turn away from handset manufacturers or Networks that had let us down in the past they WOULD learn. I suspect we get what we deserve.
As opposed to AAS geeks buying online or from eBay 8-)
The problem is essentially...money! The not-very-new-but-still-current practice revolves around the 80/20 rule. Every CTO / CFO will know that it would cost too much to fix 100% of bugs and it is generally recognised that 80% is the point where it becomes inefficient to do so. The perception is that if you keep 80% of the people happy then you can spin / fix your way to success. And actually, if we take the n95 as an example, the rule would prove to be right on the basis that the line has become one of the most successful products in mobile history.
Us geeks will always have to go through some pain, but we like that really don't we - I mean, it does actually keep some of us in a job - no? But for the average Joe on the street who will maybe only use 20-30% of the phone's functionality - if it works most of the time and they can put a "cool" theme on it or impress their friends with the interface, then they're largely pretty happy.
Even with extensive marketing, it takes months for a product like a high-end smartphone phone to achieve a decent market penetration. That gives the R&D guys plenty of time to get the fixes out. In the meantime, the sales-force are selling the phone on features that most of the public will not use in 18 months as much as a decent reviewer will use in a few days.
So all-in-all, while it's frustrating, I personally can't see it changing any time soon.
ILG
The Canalys numbers are somewhat skewed by the huge Apple numbers from a few months ago, but things should settle a little for Q4. Doubtless Rafe will chip in with his opinions soon. Or maybe we'll tackle the numbers in the podcast....
Steve
Nokia announced the N96 months ago, and now there are several negative comments about how long it has taken to arrive - and how the specs look less impressive now than they did at the start of the year.
If they had taken this time to produce a solid, almost-bug-free product, would these reviewers be less harsh? Let's be honest, it's pretty poor that after all that time it has all these (reported, I haven't used one) problems.
How about taking the N85 route - announcing the product barely a month before release; does this give them enough time to produce a decent firmware?
I was part of the beta testing for both the N95 and N95 8GB. I spotted the red tint (Mars Effect) on the N95 photos before general release, but I think it was about 4 or 5 months before Nokia corrected things in Firmware. Hey, at least they did correct it!
There's a reason why these pesky customer satisfaction surveys keep showing 90% for the iPhone, and it has nothing to do with marketing. There is only one iPhone and practically everybody at Apple has their attention focused on the iPhone like a laser beam, and it shows.
This scattergun approach to product design and product launching is going bite Nokia on the bum in 2009.
They need to take a cue from Apple on that. Take a N96/N95 that was released and compare the same with iphone.. in Plain english Iphone just works fine. Just wondering why cant they even do basic testing before releasing phones that cost bomb. Nokia has used me as guinea pig with buggy products like N95, N800 that needs continuous firmwares just to work propelry. Thanks nokia. Would never buy your product again.
@Steve, you've pulled the exact nerve that hurts, its a real issue and lets hope Nokia does something about it.
And I think Bassey has the other main reason right here:
| If Nokia (or whomever) takes too long testing a device before it is released to the networks it could be old hat. |
And if the manufacturers don't do that, people and networks may turn to other manufacturers instead. The manufacturers are in an impossible position.
And it's worth remembering that phones are unlike anything we've seen before in the computing world:
- Mobile phones sell a billion units a year so their users are utterly diverse with totally different needs and budgets. There is no typical phone user, they cover the entire human race, which isn't the case for any other computing device.
- One manufacturer alone may have to release dozens of models a year to serve all these different target audiences and get a respectable market share.
- Each model has to have dozens of variants for different frequencies and/or network operators. Add all those up and you're looking at literally hundreds of different device configurations to be tested every year from just one manufacturer, and they have to then be re-tested if new firmware versions come out.
It's easy enough to test one version of one phone with one firmware version, but that's just not a realistic mass market model in today's global phone market.
Some people wondered why Apple was restricting the iPhone to just one network in each country and releasing just one model each year, but a big part of that is because they wanted to avoid this testing headache. The downside of that is the severe limits on how well the iPhone can sell, which is part of the reason it's only got a 2% market share. If Apple is happy in that niche that's fine, but if they try to expand significantly beyond that they will have to spend less time testing each model, and have to accept a lower level of quality control.
| They need to take a cue from Apple on that. |
You think Nokia should release just one phone model a year?
And this is why I brought back a new N85 this week and went back to my e51. Although that has only half the functions, it at least almost never crashes on me, doesn't wobble around and is speedy even with several applications opened...
I am surprised that you found N85 so much buggy that you had to switch to E51.
It has worked well for me (1-2 crashes in last 2 weeks).
bartmanekul,
>>Exactly. The N85 has very bad camera noise on almost any low light situation.
True. I was not able to take evan a single half decent photograph in low light condition. But I think this will be an issue with almost all the similar mobile cameras.
|
Originally Posted by AmitKumar
True. I was not able to take evan a single half decent photograph in low light condition. But I think this will be an issue with almost all the similar mobile cameras.
|
My original got sent back for a refund, due to the GPS being faulty. It also had a loose slider.
These are all things expirenced with the N95 when it came out, and a few others.
To release a phone with bugs and issues is one thing, but to keep doing the same ones over and over....
Firmware development is normally next to stabilise then it is down to software.(when I say firmware I mean firmware as in VHDL)
In most companies software will go through peer reviews, Design Verification Testing and will not be until critical bugs are fixed.
Unfortunately due to consumer demands and the need for leading companies to be first to market means that a large amount of CVT is done (Customer verification) . From a business point of view being first to market is 1 of the most important things and this is the only way that companies like Nokia and Samsung will remain competetive and the only way we will continue to reap the rewards of their innovation.
I guess Nokia will use their Pilots program for this - geeks getting unreleased phones as beta testers might help mitigate this issue a bit ;)
| You think Nokia should release just one phone model a year? |
As for the testing: I came to the conclusion that for most tech companies, for most product categories, quality just does not pay, and going for quality borders on suicide (if you don't happen to be the one exception that confirms the rule, like Apple).
Many users just don't use their phones extensively enough to hit the bugs - the phones phone and text quite alright. Then, many other users long for a new shiny toy every year and don't let get bugs in the way of that longing. Very few people are ready to wait for a quality product and then pay some premium for it (because testers also want to be paid, after all).
i agree these devices are complicated and therefore hard to test but we pay alot for them and deserve a decent product for our money.
I am sick to death of buying something that doesn't really work. For me firmware updates should be for adding new features or fixing the odd feature not for actually making the device usable.
the fact that people actually defend part finished products is hilarious - talk about fanboys brainwashed by marketing! have some standards people!!!