Multi-touch interfaces

Getting from marginal to great

BY JON PEARSON
Cypress Semiconductor
San Jose, CA
http://www.cypress.com

Multi-touch human interfaces promise to impact the world in much the same way the first graphical user interfaces did. The race has just begun between different implementation methods, but make no mistake, it will be won soon and not necessarily by the best technology. Unless the engineering community steps in to ensure that multi-touch becomes a useful technology, we may have to live with it being marginal for a long time.

What is multi-touch?

The world applauded Apple’s first-generation interpretation of a multi-touch interface in the iPhone 1.0. The new user interface worked, offered appropriate functionality, and still made phone calling easy. The iPhone added gestures to our vocabulary, offering two-finger pinch to zoom in, two-finger open to zoom out, and two-finger rotate; all intuitive and useful on a device with a camera and small screen. Effectively, these “Multi-Touch Gestures” (see http://www.cypress.com/?id=1193) sense the position and relative motion between two fingers and translate them into a gesture to which a program can react (that is, rotate, zoom, select, and move).

Fig. 1. Example of a multi-touch display, showing three simultaneous finger readings.

Not many people realize that Apple began its foray into multi-touch with its large touchpad notebooks. Users could scroll using two fingers on the touchpad and even access right-mouse button functionality by putting two fingers on the touchpad and clicking the button (eliminating the annoying -click). Again, it’s hard to call these choices anything but useful and welcome.

Recently Apple introduced their next-generation MacBooks using the familiar two-finger gestures of iPhone with new — and less intuitive — three- and four-finger gestures. Microsoft began shipping Surface, its multi-touch brainchild that comes with a hefty $5,000 to $15,000 price tag and is clearly aimed at large businesses. Surface supports some gestures, like two-finger zoom and rotate, but seems targeted more at a multiuser experience. However, at the October 2008 Microsoft Professional Developer’s conference, Microsoft promised multi-touch support in Windows 7.

Multi-touch technology today

Legacy single-touch interfaces are most often based on resistive “touchscreen” technologies where the stylus position yields two voltages representing the same x-y data as a mouse. For their multi-touch interfaces, Apple uses capacitive sensing while Microsoft’s Surface uses cameras. Capacitive sensing is also the technology typically used in single-touch trackpads in notebook computers today. There are several module, screen, and individual silicon vendors supporting multi-touch and, depending on the technology, it is possible to simultaneously sense the position of all 10 fingers on a display (see the video here: http://download.cypress.com/2_0/ui/flash/video.swf).

The challenge the industry faces is that there is yet no one standard way to interpret multi-touch data or implement them in applications. Consider the two-finger gestures popularized by the Apple iPhone. For these useful gestures to penetrate the lexicon of every laptop, every program needs to change to accommodate them as well as the operating systems we use.

Beyond intuitive two-finger gestures, the picture quickly blurs. “Multi-Touch All-Point” technology enables many simultaneous inputs from the touchscreen or touchpad. There are huge opportunities here, but even Apple and Microsoft are not exploiting them to the fullest.

Apple has included three- and four-finger gestures on the new MacBook, but only Apple applications use these gestures. Additionally, the thought process behind which functions should take three fingers and which four fingers appears to be arbitrary. What doesn’t make sense is the comeback of the button, being used as a modifier for three-finger gestures even though the touchpad is large enough and has the ability to sense more fingers. The situation becomes reminiscent of the trouble Palm PDA users had trying to master Grafitti.

Intuitive gestures equal easy adoption

Likewise, Microsoft with Surface seems to be stumbling in the dark when it comes to using more than two fingers. For instance, Surface’s virtual air-hockey game demo has players grab and move their controllers with a single finger. In addition to feeling awkward, users find the controllers slides around as if they were really using only one finger to control a large disk. There are other problems too, such as the controllers sometimes switch players when they get too close.

Fig. 2. Single-touch “Double-Tap” selection gesture applies to multi-touch displays.

This commentary is not intended to denegrate Apple’s MacBooks or Microsoft’s Surface, but rather to highlight the fact that the multi-touch field is wide open. The technical capabilities available today can offer a more natural and intuitive user interface but if and only if we as engineers harness and manage that power with care.

The development tools for Microsoft, Apple, and Linux all provide built-in, standard support for keyboards and x-y pointing devices (mouse and trackpad buttons as well). As long as the “input” from these devices is translated into one or more keyboard keys and x-y position, any application can use that input. The question for developers of multi-touch interfaces is what the development community needs to do to secure a similar level of freedom for multi-touch input devices and the programs that interact with them. Specifically, we need a standard set of capabilities so program developers do not have to worry about input devices and input device makers do not have to worry about applications. The answers we develop today will seriously affect the future of multi-touch interfaces.

The future of multi-touch is in our hands

“Standards” for multi-touch interfaces are already being developed and so now is the time to get involved and help shape the future of multi-touch. Let me put forward some of my suggestions for a better multi-touch future. Read them, build on them, and get involved by either working with the standards-setters or by putting multi-touch device into people’s hands so we can mobilize more troops in this fight for a great multi-touch future.

Fig. 3. Intuitive “Zoom-In” gesture with two fingers.

First, we need a few standard, intuitive gestures, followed by a standard data interface to provide position data for up to 10 fingers. Standard gestures should cover the most common operations like scroll, zoom, select, and grab-and-move, as well as all the new functions multi-touch will enable. Note that we do not need a long list of standard gestures because then users will need to print out the list and paste it to the back of their devices, just like we all did with Grafitti on our Palm PDAs before we stopped using them.

We also do not need multiple company-patented sets of gestures. Rather, innovators should patent better and smarter techniques of recognizing gestures. Gesture detection cannot be forced exclusively on the operating system nor can it be forced exclusively on the input device. The best standard approach would accommodate both for maximum flexibility. However, if a choice between operating system or input device is forced, the operating system must be allowed to win.

Do not stop with standard gestures nor define everything as a gesture. Also, define a data standard for tracking up to ten independent inputs with the realization the use of such data for quite some time will be application-specific. That said, the best behaviors (we hope) will be adopted into the operating systems. Why not more than 10 inputs? Most devices are primarily single-user or have multiple users doing simpler actions on a relatively small screen. Let the special-purpose, large-format devices like Surface explore the use of more than ten inputs, and when something proves itself useful, it can trickle down.

The multi-touch train is leaving the station. For those of you who do not like everything you have seen so far, get involved and push the envelope further. Use multi-touch in new and interesting ways and let others see and hear about your successes and failures (especially the folks in Cupertino and Redmond). The industry can make multi-touch great, but only if we work together. ■