Regular readers may know that, in the interests of knowing what to recommend, I'm fond of device advocacy, i.e. comparing the pros and cons of different phones to try and come up with some kind of ranking. Usually these pieces are centred around specification or feature lists, not unnaturally, but here I'm going to take a dozen real world tasks, things that you or I do (or would like to do) every single day, and then assess how easy and/or quick these tasks are to perform.
For the purposes of this comparison, I'm going to take the six month old Nokia N95 8GB, typical of the current S60 range, the brand new Apple iPhone 3G and the also-new HTC Touch Diamond. If Apple (and HTC's) marketing is to be believed, the iPhone should be an easy winner in a usability test, but the extra maturity and featurettes in S60 3rd Edition also make it a hard act to follow.
NB: just to head off a flood of emails, yes, I know the Apple device photographed and screenshotted here is the iPod Touch and not the iPhone 3G - but the latter is currently rarer than hen's teeth and Apple's policy of keeping the Touch bang up to date with the iPhone 2.0 software means that, for most purposes, the two are nigh-on identical. I've used various 'real' iPhones in the past few months during the course of research, fear not, so I've got a good handle on the differences...
As for the real world tasks, I brainstormed them with the rest of the AAS team, working out the most common things that we all wanted from a phone. They had to be non-trivial (i.e. making phone calls was out), non-geeky (so no requirements for VNC clients) and not obviously biased (e.g. explicit multi-touch photo manipulation or VGA video capture and editing). I've tried to be as fair and open as I can. For each task, there are 20 points up for grabs for each device, split between accomplishing the task, speed and efficiency and any other relevant factors.
1. Set an alarm for the next morning (and for the rest of the working week)
Nokia N95 8GB:Eleven keypresses (depends on where you've set 'Clock' to appear, though!), around eleven seconds to set the new alarm up, including the workday repeat. Score: 19 (loses a point for the user having to work out to press 'right' to get to the 'alarms' tab)
Apple iPhone 3G:Eleven screen taps this time(!), also (amazingly) taking around eleven seconds to set up. Score: 19 (loses a point for not knowing about workdays and the user having to manually tap on each day required)
HTC Touch Diamond:Nine keypresses, taking around seven seconds. Score: 20 (tempting to take off a point for a fiddly old-school interface, but this is counter-balanced by the general speed of access)
2. Enter a new calendar entry "Meet Jane at gym, take present" for 6pm the current day, with an alarm
Nokia N95 8GB: 27 seconds. Score: 18 (no real problems, but some words needed multiple taps to get to the right word predictions, plus the alarm setting needed quite a bit of scrolling down the dialog)
Apple iPhone 3G: 36 seconds. Score: 15 (multiple dialogs for title, timing and alarm slowed things down. Text entry was as fast as using T9 predictive text on the N95 keypad)
HTC Touch Diamond: 31 seconds. Score: 16 (slowed slightly by pause for getting the stylus out to use the fiddly Windows Mobile Calendar interface and qwerty on-screen keyboard, but speeded up a lot by the way alarms are 'on' by default)
3. Check your GMail inbox, view a message and fire off a reply
Nokia N95 8GB: 50 seconds using built-in Messaging, accessing GMail via IMAP push email, slowed by the speed of Messaging and the clumsy access point selection etc. But only 30 seconds using Google's Java-based GMail client. Score: 14 (i.e. not a great experience, despite the true push nature, and almost have to use the supplied Google utility)
Apple iPhone 3G: 25 seconds, using GMail via default iPhone IMAP access. Really fast and slick, plus great presentation throughout. Score: 19 (let down a little by not being true IMAP-push, you have to refresh the mailbox manually)
HTC Touch Diamond: 70 seconds. Appallingly inconsistent interface, often unresponsive, slow message header downloads, terrible text entry practicalities. Score: 9
4. Write a 25 word text note
Test text: "Aiwa VCR bought at Comet on 20/7/08. Turned it on to see a blue error message. Returned to the shop and am awaiting a replacement."
Nokia N95 8GB: 93 seconds. I'm not the fastest texter in the world, but not that slow either. A good average score: 16
Apple iPhone 3G: 75 seconds. The iPhone's intelligent keyboard is a wonder to behold. Taking things a step on from predictive text, it allows for complete keyboard mishits and works out, based on the approximate location of your taps and their order, what word you were actually going for when you typed "rrtot" (error). A good, though finger-exhausting, alternative to a real thumb keyboard. Score: 19
HTC Touch Diamond: 156 seconds. I could probably have done it slightly faster using the tiny one-letter-per-key keyboard, but heck, I wanted to use at least one of the features of the new HTC software suite. This time was with the 'Compact qwerty' keyboard, which does well when plain English is needed, suggesting good words, but is incredibly slow when fiddling real names, number or symbols are needed in-line. Score: 10
5. Look up a UK train time, Reading to Torquay (a real journey that I'm doing in the near future)
Nokia N95 8GB: 55 seconds. Typed 'Rail enquiries' into Google, got automatically through to the mobile web site, the query was handled very quickly and smoothly. Score: 16
Apple iPhone 3G: 90 seconds. Same route, but the iPhone's version of Safari get served up he far more clumsy desktop version. Still navigable, but.... well.... clumsy. As a side-note, I noticed that one of the original iPhone 'web apps' (akin to Nokia's 'widgets' or 'web runtime' apps) offered a UK rail lookup service. With a shortcut to this on the iPhone homescreen, the process took a far more competitive 50 seconds. Score: 14 (a user would have had to have had a good rummage in the web apps scene to know about the Kizoom widget etc - but hey, at least it existed)
HTC Touch Diamond: 130 seconds. Same route, with Opera 9.5 also being served the full desktop version of the rail enquiries site, which slowed things down. The initial search for the site and the entry of station names was also hampered by one of the new keyboards and I had to switch to the old, tiny version in order to see the data fields properly. Score: 9
6. Find an ATM nearby and plan a pedestrian route to it
Nokia N95 8GB: Using the built-in Nokia Maps, 39 seconds. This was using the supplied 'Banks/ATMs' points-of-interest ('Places') category. Every local match (there are quite a lot, it even included places which didn't have an ATM but which would give me cash etc.) was shown. Picking a match and choosing 'Show on map' brought up the graphic shown here. 'Walk to', i.e. pedestrian routing is built-in, as is voice-driven car navigation, provided you're subscribed (which costs money). Route calculation was slowish, but, once calculated, navigation was flawless and real time. Score: 17 (competent and complete, without being jaw dropping)
Apple iPhone 3G: Using Google Maps (built-in on the iPhone), 27 seconds. I did get it to find me all the nearby cashpoint machines - this was after I got no sensible matches for "ATM" or "Cashpoint" - it turned out that "Cash" was what was needed. Tapping on 'Directions' and then 'Start' brings up the first turn instruction, but it's not real-time driven. Not a problem for a pedestrian to keep up, though. Score: 16 (it was fast and the map-based directions work fine when not in a car)
HTC Touch Diamond: Using Google Maps, built-in: 56 seconds. Despite the built-in QuickGPS (like Assisted GPS), GPS lock was slower than on the other two devices, and Google Maps interface under Windows Mobile could be slicker. Still, it got the job done, with the same manual advance turn issues as on the iPhone. Score: 15
7. Rip a new CD and get it playing on your phone while you go running
Nokia N95 8GB: Like all Nokia Nseries, plugging in the USB lead gives the choice of 'Windows Media' mode, upon which adding music to your memory card is as easy as adding albums or playlists to your 'Sync' list. However, transferring music files is slow, at around 1MB/s and there's an extra delay while the N95 refreshes its music directory. Playback is great with both d-pad control and dedicated music keys. Time taken from starting to rip to jogging along, listening to Deep Purple on wireless (A2DP) stereo headphones: 4 minutes. Score: 15
Apple iPhone 3G: Apple built its music device empire on seamless music handling and ripping and syncing a CD of music (to unencrypted AAC format) is as easy as you'd expect. Ripping was slower, but music transfer speed is three times faster than for the Nokia, the two factors cancelling each other out. Playback is via 3.5mm wired headphones only, as the iPhone doesn't support A2DP (wireless stereo). Total time taken was 5 minutes. Score: 13
HTC Touch Diamond: Working along the same lines as the Nokia N95, the Touch Diamond integrates neatly with Windows Media Player which, although a poor relative of Apple's iTunes in terms of online store integration, is a lot quicker for local operations. Music transfer was also around 1MB/s to the Diamond, with similar overall times to the N95 8GB. And A2DP works, too, although the confusing duplication between the TouchFLO music player and the built-in Windows Mobile Media Player loses the Diamond a point for simply being inelegant. Score: 14
8. Take a photo and email it to a relative
Nokia N95 8GB: As simple as it gets, honestly. Press and hold the camera button to launch the camera. Focus and shoot a 1MB 5 megapixel image. Select 'Send|Via e-mail' and press in the d-pad to select a contact. Done. 33 seconds from starting the camera to the email starting to send. Score: 20 (Perfect)
Apple iPhone 3G: Ah. The first of the iPhone's real limitations really hit home here. Firstly, the camera is only 2 megapixels and with average optics. Rather more worryingly, the emailing process reduces every image to a maximum of VGA (0.3 megapixels), complete ruining what was already only an average photo. Time taken from starting Camera to the email starting to upload: 23 seconds, so at least it's quick. Score: 7. (Interestingly, and even more worryingly, if you save a large (say 3 megapixel) image off the web and then try viewing it in Photos, the software goes loopy, the screen corrupts and the iPhone simply can't handle such a large image, let alone getting as far as an email dialog. A big OS update needed, Apple.)
HTC Touch Diamond: Pretty much as with the Nokia N95, except that the user experience is much more clumsy. Camera has to be launched from a fiddly on-screen button and the email recipient can only be picked after using a menu option. The full image is sent, of course, but the camera itself is a mediocre 3 megapixel non-flash unit, although at least there's a semi-automatic focus. Time from start to email send: 35 seconds. Score: 14
9. Shoot 1 minute of video of your kids and dog in the park
Nokia N95 8GB: Near perfect again. Sorry if I sound like an N95 fanboy, but come on - VGA resolution video at getting on for 30 frames per second through semi-professional Carl Zeiss optics. Video is captured in H.264 MP4 and can be uploaded to YouTube from your PC or even to Ovi (for example) directly from the handset (probably via Wi-Fi, though, at 20MB per minute of footage!!) Score: 19 (loses a point for occasionally dropping the odd frame and thus appearing jerky)
Apple iPhone 3G: Next. There's no support at all for shooting video from the iPhone's camera. This is a showstopper for some, I'm sure. Sorry.... Score: 0
HTC Touch Diamond: Not in the N95's class on any front, but the Diamond does at least shoot video. In fact, it's at CIF (VideoCD) resolution and even sports initial focus in video mode, but it's downhill from there on, with poor picture quality, sub-par frame rate and truly appalling recorded sound. Add in a confusing storage system for your videos and you end up with a score of 10. And that's being generous.
10. Browse through a couple of subscribed RSS feeds in Bloglines
Now, I know that there are a bazillion RSS readers out there (if you don't know what RSS is, then you're utterly missing out on the fastest way to stay up to date with all your favourite web/news sites), but by far the two most common are Bloglines and Google Reader. Both of which are web-based, meaning that you can read stories and have what you've read marked in perfect sync, whether you're reading on a phone or desktop or set-top box. Anyway, I use Bloglines, so this was my test case.
Nokia N95 8GB: With Bloglines Mobile bookmarked, the N95's QVGA screen shows feeds at super-legible size, although once into each feed there's lots of scrolling needed. Score: 16 (no showstoppers)
Apple iPhone 3G: Safari, in its infinite wisdom(!), decides to show Bloglines Mobile in a non-zoomable, microscopic way. Even worse, the links simply don't work - very disappointing. Never mind, there's always the full Bloglines interface. This appears at first glance just as on the desktop, which is good, despite the amount of vertical scrolling needed to read down busy feeds (e.g. Engadget). However, because the Bloglines 'framed' interface doesn't work on the iPhone, you then have to scroll all the way back up each page in order to get back to your feed list - at least there's a half-remembered Safari shortcut, in that double-tapping at the top of the screen zooms you back to the top of the page. And then I vaguely remembered something about a test interface for the iPhone, i.bloglines.com - which works brilliantly. Phew. Score: 14 (we got there in the end, but the average user would struggle to find out about the optimised interface)
HTC Touch Diamond: Remembering the iPhone experience, I went for the full Bloglines interface first and this worked but with the same scrolling caveats and without the handy 'top of page' shortcut. I switched to Bloglines Mobile and this worked as on the Nokia, but with an annoying necessity to have to double tap the feed list every single time to zoom in to read it. Score: 14 (at least feed contents were scaled properly and were easy to read down the page)
11. Subscribe to an audio podcast, download it and listen to an episode
Nokia N95 8GB: A system of 'Web feeds' is integrated into S60 Web, it's all rather primitive and low-tech, but it works. You click on a RSS link in a web page and the feed gets added to your feeds list. Clicking through to a feed refreshes it and, if it's got media attached, you can click the link at the bottom of an entry to download and play it. (Worth noting and trying for top level pages, there's also 'Subscribe', the very first menu option, that appears when Web has detected that there are RSS feeds associated with the page.) Fairly quick and painless, though no glitz or glamour. Score: 17
Apple iPhone 3G: Navigating to a page with a podcast's RSS link and tapping on it links through to a reader.mac.com URL, presumably to handle the RSS feed online, rather than locally on the iPhone. I tried it numerous times and just got '.Mac Reader Error', as shown here. I'd blame server congestion, but I was trying (9am GMT) while the whole of the Mac-loving US was still tucked up in bed. I'm guessing the .Mac server would show the feed's contents and offer a download link for any audio podcast content, but.... I tried several other sites but found it impossible to do anything sensible with a direct RSS link (not helped by the lack of copy and paste on the iPhone). "Stop!", you say, "this is what iTunes is for!" And you're right. Finding podcasts, subscribing, downloading and syncing them via iTunes when you're back at your home/office is a piece of cake, so I'm not going to hammer the device too much on the podcast front - but wouldn't it have been nice to have some direct handling as well? Score: 15
HTC Touch Diamond: A surprisingly seamless experience for anyone with their wits about them. Clicking on an RSS link on any web page offers the chance to copy it to the Windows clipboard, then you just have to 'spot' the new RSS Hub application in Start|Programs. Opening this and adding a new channel then automatically inserts the copied URL from the clipboard and offers to download the feed's official name. A couple more taps and I was listening happily. Score: 17
12. Find a video of "Abba - Dancing Queen" on YouTube and watch it.
Phew - A little fun after all that RSS hard work!
Nokia N95 8GB: Heading to www.youtube.com is hard work on S60 Web - it's a high bandwidth page and rendering seems to take an age. It works though, and the N95 8GB just about handles the in-line Flash video, as shown. Playback is a little jerky at times, but watchable, full-screen. Time taken to get watching: 2 minutes 5 seconds. Going via YouTube's mobile site failed miserably, with the streaming 3GP version failing to get past my Wi-Fi router. Plan C is YouTube's very own S60 client, Java-based - this installed fine but then (despite correct RealPlayer settings) it too failed to stream its video through my router's firewall. Plan D (you can see where this is all going, can't you...) was to fire up the third party freeware Mobitubia, which managed to get Dancing Queen playing on-screen in 48 seconds from a cold start. Impressive, but - again - your average user may not know about such a useful piece of software. Score: 12 (uninspiring without going third party)
Apple iPhone 3G: Wow, the advantage of a fully configured, built-in client, with a glorious user interface. 40 seconds from a cold start to hearing Anna-Frid and Agnetha warbling. Video quality was top notch too. Score: 19 (one point lost for the catalog of videos not being complete - not every video on YouTube is here, sadly)
HTC Touch Diamond: Another built-in, purpose-built YouTube client, similar to the iPhone one but slightly slowed down by the text entry limitations for my search string. On the plus side, it queries the full YouTube (as far as I can tell) and it only took 46 seconds to start playing from a cold start, on the downside, playback was a jerky in places - which shouldn't happen for a native Windows Mobile application. Score: 18
Final scores
Nokia N95 8GB: 199 (Max score: 240)
Apple iPhone 3G: 171
HTC Touch Diamond: 166
Conclusions
Yes, this is a Symbian-focussed web site and I'm sure a few people here will cry foul, but let's examine the results more closely. The tasks that the AAS team set really were everyday requirements and not biased in any one direction. The N95 8GB comes off best by a margin of 28, as the best all-round (smart)phone for the user who wants more than just phone calls and texts. But what's fascinating is that this margin of 28 would be wiped out if the iPhone had had a decent camera that shot video clips. In other words, camera-apart, there's very little to choose between N95 8GB and iPhone in terms of real world speed and functionality.
Of course, this being the real world, the iPhone's average camera and lack of video leave it scoring just more than its main competitor (at least, if you believe HTC's marketing), the Touch Diamond, which manages to score less despite having a reasonable camera and a semblance of video recording. I've shouted down the Diamond before, in Smartphones Show 63, and I'll do it here again. It's a powerful Windows Mobile 6.1 business handheld shackled by a clumsy, slow and inelegant add-on UI and confusing set of text entry mechanisms. The iPhone, on the other hand, manages to impress for much of the time and it was a genuine pleasure to pick it up and try and do one of the tasks above, even if - like the photo emailing task - the results themselves were less than impressive.
But I have to emphasise once again that, as a 'swiss army knife' device, the Nokia N95 8GB is the best all-round phone in the world - bar none. It's not the thinnest or the sleekest or the fanciest or the fastest or the newest or the most trumpeted, but it can do everything you ever need it to, without breaking much of a sweat, and it has very, very few weaknesses.
Steve Litchfield, All About Symbian, 22 July 2008