The 'full web' still unattainable for many when mobile - Help needed!

You've seen the 'pinching and zooming' adverts for many (non-Symbian) smartphones, showing lightning fast manipulation of full desktop-class web page renders, with new pages 'coming down' in a matter of seconds. "It's the Internet in your pocket" say the promos. And, from my own observations, for many people this is utter pie in the sky. Out in the real world, mobile coverage and bandwidth falls diabolically short - which partly helps explain the popularity of a certain proxy-based web browser that works on everything and enables not the 'real web', but more 'looks and feels a lot like the real web, but isn't really'...

In these days of iPhones and huge-screened Android smartphones, there seems to be an expectation in the air that their web browsers can, aside from a few Flash hiccups - depending on model, give you access to the 'real web'. The 'full Internet'. By which the ad writers, marketing men and, indeed, journalists mean that what you see on your 3.7" or 4.3" high resolution screen is more or less the same as you'd see if you logged in with a Windows, Mac or Linux desktop, running Chrome, Internet Explorer, Firefox etc.

And, barring a few concerns over trying to view 800 or 900 pixel-wide pages on a 4" screen - after all, zooming in is usually easy and then you can see areas of text much more easily, the Webkit-based browsers in the iPhone, Android and (yes) Symbian-based smartphones do a pretty good job of laying out pages, complete with images, adverts, links and (on Symbian and on some other devices) Flash animations and videos.

Here's the promo scenario. I'll assume you've got a shiny new Nokia N8 (or iPhone or HTC Desire, but let's keep things Symbian here since this is AAS, after all) and are seated comfortably in a coffee shop. With free Wi-Fi on tap, you're able to surf happily, with pages appearing in ten seconds or so and with multi-touch giving you an easy way to zoom in when needed and interact with each page. Job done.

Here's another common promo example. You're seated on a train, on the 20 minute commute into the city, and you're again surfing happily on your Nokia N8, using the 3.5G signal to access web pages in reasonable time (15 to 25 seconds to be fully rendered). There's the occasional drop in download speed as the train moves through cell boundaries, but on the whole it's a pretty satisfying web browsing experience, considering that you're on the move in public transport at a steady 50mph or so.

Then there's the real world.

The UK is thought of as pretty well covered by cell signals. I'm on Vodafone, but most of the other networks also have something like 99% coverage, meaning that you can, more or less anywhere, make a phone call. And no, this isn't a build-up to an anti-iPhone 4 joke. But, and this is a big but, just because you can make calls does not mean you've got a full Internet service.

Wi-Fi download speeds on a typical café network are usually around 1MB/s (Megabytes per second) for a high spec smartphone in real world conditions. 3.5G speeds often average out at 400kb/s (kilobytes per second). 3G averages about 100kb/s on a good day, while EDGE gets you an average 15kb/s and bog standard GPRS brings up the rear with about 4kb/s *

* note that I've used data transfer per second in bytes, not bits. Other often inflate figures by referring to numbers in Mbps or kbps, where the 'b' is bits. There are, of course, 8 bits in a byte, though a factor if around ten usually applies, once you've factored in error correction, parity, headers and so on!

So that's a maximum of a Megabyte every second under good, promo-worthy conditions. And a minimum of 4kb under worst conditions. (Well, worst conditions would be no coverage/signal at all, but let's assume that you've got a few bars on the ol' meter, at least.) In other words, two hundred and fifty times slower!

Web pages in 2010 are far more than simple HTML scripts with a few images to liven things up. From cascading style sheets to annoyingly large blocks of javascript code to non-optimised images to unnecessarily bloated HTML to infuriating slow-loading Flash, web pages are complex entities. Where a web page circa 1995 averaged 10kb, or 50kb with images, pages regularly clock in at 100kb of HTML and scripting, plus another 1MB of images and the odd 400kb Flash object. The average download needed in 2010 to grab a web page in full is now well over a Megabyte, sometimes two or three Megabytes. For one page!

So you can see that even the top-end, high spec 2010 flagship phone (e.g. the N8, or iPhone or HTC Desire) is going to be doing very well to download, interpret and render well over a Megabyte of content in ten seconds or so on Wi-Fi and 25 seconds or so on 3.5G data. Putting this feat into perspective though is the fact that the same phone, grabbing the same page over a lowly GPRS data connection would take over four minutes.

I had personal experience of this enormous real world bottleneck when heading out to the South West UK this week. There was some coverage everywhere I went, usually showing two or three 'bars' of GSM at least, but never 3G. GPRS connections were most common when I picked up email and got nicely synced up with the world via Twitter/Gravity, though I occasionally got upgraded to EDGE. For these tasks, even GPRS was practical - after all, neither emails nor tweets involve huge amounts of data transfer and there was little problem with any kind of latency.

The problem came when, in an email or tweet, a web URL was mentioned. As I'm accustomed to doing back home in Berkshire, I clicked on the link and Web was duly launched. I then had to sit there watching a white screen with small black loading bar, watching the byte count rise. 100kb.... 200kb.... 400kb.... over a minute into the page load and the screen was still white. At this point I gave up. This experience was then repeated three or four times over the next few days. Each time hoping that the web page I had been redirected to (usually via a URL shortened address, so it wasn't possible to see the exact site being referenced) was lighter and better designed than the previous ones. To no avail. I simply could not access these modern 2010 web pages on a non-3G connection, it seemed.

I know I'm the exception here, but my 3-Lib site's pages are (almost) hand-crafted HTML and the average page size is around 20kb, with an average of 50kb of graphics. (Yes, I learned my craft in days when every kilobyte made a difference and I'm effectively a dinosaur, but surely there's a case for these skills to make something of a comeback?) Over the same ultra-slow GPRS connection, most pages on 3-Lib came up in about ten seconds. 

But back to the 2010 'web'. The truth is that most sites are designed using WYSIWYG or wizard-based tools far removed from the original raw text tags. They also make heavy use of interactive code and graphical effects - all of which are fine when there's bandwidth to spare but which make no sense whatsoever when you're limited to under 10kb/second. And it's not just the UK countryside - I'll bet that far higher proportions of the USA (for example), being a far bigger country are left with even less signal - often none at all [which at least would be less frustrating, since you wouldn't even try to get online!]

So the popular promo use cases mentioned above, for large-screened, Web-centric smartphones, simply fall down flat once you're away from Wi-Fi and away from urban 3G data. One solution, of course, is to restrict yourself to just the 'mobile' web, using pages optimised for low bandwidth, and I explored this in depth in my feature There's a bookmark for that! recently.

Another solution, and the one I resorted to in Somerset, desperate to get hold of a few specific web pages but unable to afford the tens of minutes I'd need to load them in Web over GPRS, was to use the rightly famous and much-plugged around these parts, proxy-based browser Opera Mini. See our Opera Mini tutorial, for example. The idea, as you'll know by now, is that Opera's servers do all the heavy lifting mentioned above (1 to 2MB of content), rip out all the bits you won't need (Flash, javascript, etc), compress what's left and then squirt a far smaller (typically about 50 to 100kb) set of files at you. The interface isn't much different from a traditional web browser accessing pages directly, and yet the bandwidth needed to access exactly the same pages is some 10 to 20 times less. If you haven't already got Opera Mini installed on your phone, it's in the Ovi Store and it's also directly downloadable at mini.opera.com.

In an ideal world, one could argue that bandwidth would be ubiquitous and both the mobile web and proxy-based browsers shouldn't be needed. As an old-school coder myself, I weep at the flagrant inefficiencies in modern web pages, yet realise that I'm the lone voice in the wilderness - the Internet has become synonymous with broadband, multimedia extravagance and complex interactivity, most of which means page bloat of one kind or another.

But don't let's pretend that this can all translate to a similar experience on the phone. However large your screen, however fast your processor, you'll come a cropper when you hit the city limits and are at the mercy of cell tower range and the humbling laws of physics.

Thank goodness for Opera Mini.

Steve Litchfield, All About Symbian, 2 August 2010