Touching is beleiving:  The Perceptual Advantages of Tangible Interfaces

We have always prided ourselves in bringing top-quality experts to share with us the latest and greatest in any given domain of computer graphics or interactive techniques. This time, we have outdone ourselves, and the congratulations go two of our speakers tonight, both Canadians, for having been awarded the Best Paper Award at the ACM SIGCHI Computer-Human Interactions Conference taking place this very week here in Montreal. Indeed, it is in conjunction with that very conference that it has been possible to bring together 4 experts from three different countries on the same evening. ACM SIGCHI is actually a sister organization to ACM SIGGRAPH, and there is a good deal of overlap and exchange between these two communities, as the interest these speakers have in coming here tonight demonstrates. We appreciate them taking time after a busy conference day, and look forward to having our intellectual horizons broadened!

An interesting coincidence occured to me during the organization of this event, and it has brought me much insight into the nature of tangible interfaces: I broke my finger. Not just any finger, but the all-important index finger, and on the hand I normally write with! A most unfortunate event, to be sure; nevertheless, it has at least led me to appreciate the extent to which we depend on this traditional tangible interface in our everyday lives.

In the words of ergonomic design, or to paraphrase Karon Maclean, one of tonight’s speakers, the “complex set of affordances” of my hand have been severly compromised! Every minor task that relies on manual abilities ( and there are a lot of them, believe me!) has become an opportunity for me to appreciate the wide adaptability, reliability and extreme versatility of the human hand as a means of interacting and communicating with the world around me. ( Ed. note: the author’s ensuing digression on the growing role of fingers in digital-era communication has been seperated into a separate essay entitled “A broken finger is a speech impediment”)

Given the extreme versatility and expressiveness of the human hand, it is quite natural to consider novel ways in which to leverage its potential in the design of human-computer interfaces. Tangible interfaces provide us with an opportunity to consider not only a whole new area of interactive techniques, but also a new way of thinking about interactivity itself.

Computer graphics has always been the primary mode through which we perceive information stored in the digital domain. This is natural, because elaborate information structures, such as modern desktop applications or web interfaces can be easily comprehended with a single glance of the eye. The pinnacle of computer graphics technology, real-time rendering of 3D environments, remains, of course, the most exciting and dramatic demonstration of the supremacy of the visual mode of perception in human-computer communication. “Seeing is believing”, as they say.

Well, I invite you to consider why “touching is believing” is an even stronger assertion. If you think seeing a ghost would make a strong impression on you, then imagine the psychological impact of being touched by a ghost! Our sense of touch goes much deeper into our sense of relationship with the outside world than does our sense of sight. In this analogy the ghosts are, of course, the real-but-not-real constructs of the digital domain.

Our sense of touch is also connected to a set of information-processing mechanisms that are very different from the way we process visual information. Visual perception is excellent for analytical, "left brain" comprehension, whereas touch and bodily interactions tend to stimulate intuitive, instinctual right-brain activity. When both modes of interaction are engaged simultaneously, a much richer experience is made possible. There is a far greater opportunity for learning, or increased productivity, or long-term memory retention, depending on the context in which such interaction occurs.

We often appeal to the value of "hands on" experiences in education, for example. In the classroom, most information is communicated visually. At some point students experience a state of saturation, in which they feel that their heads are too full to absorb any more information. An evocative image is that of a CPU running at maximum capacity. Interestingly, if I were to invite those same students to manipulate a demo or apparatus in their apparently “saturated” state, they would actually be able to continue absorbing and learning in this new manner. This suggests that a better cognitive model would be that of multiple CPUs running in parallel, with each one dedicated to processing and integrating information from a specific mode of perception. The strategy, then, is to take advantage of the excellent built-in parallelization of our perceptual circuitry. We can increase the information absorption and retention through better load-balancing, in which the load of information is distributed across multiple perceptual modes.

One area within computer graphics, in which this strategy is alread being exploited is the use of audio, not only to heighten and enhance the visual experience but also to communicate information unique to this perceptual mode, such as the sound of footsteps behind you, the spin of tires on a slippery surface, or reverberation effects to suggest a large space. The principle can be extended to a much wider range of applications. Indeed, David McGookin will be touching upon some of these novel applications of audio to the visualization of mathematical graphs, of all things, in his talk later on tonight.

In a previous talk, I have referred to an emerging “plateau” within the computer graphics industry, in which visual quality, approaching the level of photorealism, just can't get much better. This will drive the industry to look elsewhere than mere polygon-counting and scene- complexity advances to sustain their business models. One direction is the use of nonstandard projection surfaces, such as the multi-projector configuration that Dr. Roy presented to us 3 years ago, or the amazing 360-degree  hemispheric projection environment that Luc Courchesne presented to us just last year. Another direction that I had mentioned is improvement of behavioral realism, through better physical, psychological, and collective-behavior models.

Both of these areas have one fundamental weakness: they still appeal exclusively to the visual mode of perception. Sooner or later, no matter how excellent or engaging the visual stimulus becomes, we will be confronted by the impasse of overstimulation. The so-called visual-effects “eye-candy” has got to have a higher and higher sugar content, until finally, it just doesn't taste good anymore. Inner experience is multi-dimensional and multi-modal: No matter how cleverly we make the visual cortex dance, it will never satisfy the mind’s hunger for a well-balanced meal containing input from a variety of senses. The rest of my body is hungry for stimulation and, more fundamentally, for participation in the eyeball-dominated experience.

Computer graphics must not lose sight of this larger emerging picture. Our hands are a key mode of engagement, not just for manipulating, but also for perceiving. Our hands can absorb and interpret information at very high rates, and the information digested in this manner is integrated into our overall sense of awareness in a unique and important way. We must balance the perceptual load across multiple sense-organs. Let the intelligence of the eye be complemented with the intelligence of the hand. The message that the tangible interfaces community has to offer the computer graphics community can perhaps be compactly expressed in a new motto: “Less eye-candy, more finger-food!”