At 100ms latency, says Paul Dietz, Assistant Director of Microsoft Applied Sciences, the “analogy of moving a real physical object breaks down”. Here's the video from Microsoft Research:
I have to say that such low latency is probably more relevant on a tablet (e.g. the iPad), where the pretence is often made of interacting with pseudo-real objects.
On a smartphone, I've only occasionally been struck by lag - usually when entering text and something else processor-intensive is happening in the background. The good news is that processors continue to get faster and faster, meaning that latency will keep on decreasing and that background tasks will also complete more quickly - though I very much doubt we'll get close to 1ms latency for all screen actions within the next five years.