Time for another Creativity Knowledge. Today I’m pointing you towards Ubiquitous Computing (Ubicomp), aka. Calm Technology.

How could you make shopping for food easier?

Imagine making a shopping list on your computer. As you head out the door to the supermarket the shopping list automatically stores itself on your mobile phone. Of course you’re always forgetting to buy milk. So your phone talks to the fridge and makes sure you’ve enough milk for the rest of the week. When you walk into the supermarket your phone gives the shopping list to the shopping trolley you’ve grabbed. Now you can easily see your shopping list on a small screen built into the trolley. As you put items into the trolley they are removed from the on-screen list.

In the meantime the trolley has talked with the shop and figured out the optimal route to get around the shop with the least congestion and fastest time. As you push the trolley around the trolley wheels subtly vary resistance, so it becomes easier to move the trolley in one direction or another. By dynamically varying wheel resistance you are unconsciously guided in different directions, such as towards a special offer and away from paths other customers are moving along.

Your shoes have also downloaded a layout of the store. While you walk around the height and softness of the shoe soles varies subtly enough that you don’t consciously notice, but they lean (and maybe lead) you away from the sweet and fast food sections. Yep, your partner has told your phone to tell your shoes that you are on a diet!

The above design scenario captures many of the ideas of Ubicomp. Background non-intrusive technologies making your life easier by weaving themselves into the fabric of everyday life until they are indistinguishable from it (from The Computer for the 21st Century).

While there is much to admire in the Ubicomp vision I often dislike one possible implication: We may become automatons of clever technologies that guide, steer and influence us “for our benefit” without us being aware of what is happening.

Mark Weiser laid out the original vision for Ubiquitous Computing in The Computer for the 21st Century, and in the essay he co-wrote with John Seely Brown, Designing Calm Technology. Both essays were, and in many ways still are, an inspiring human centered vision of the less-traveled path I (Mark Weiser) call the “invisible”; its highest ideal is to make a computer so imbedded, so fitting, so natural, that we use it without even thinking about it.

Mark identified Ubiquitous computing as the third wave in computing, just now beginning. First were mainframes, each shared by lots of people. Now we are in the personal computing era, person and machine staring uneasily at each other across the desktop. Next comes ubiquitous computing, or the age of calm technology, when technology recedes into the background of our lives.

His introduction to Designing Calm Technology convincingly describes an installation art work that embodies what he envisioned:
Created by artist Natalie Jeremijenko, the “Dangling String” is an 8 foot piece of plastic spaghetti that hangs from a small electric motor mounted in the ceiling. The motor is electrically connected to a nearby Ethernet cable, so that each bit of information that goes past causes a tiny twitch of the motor. A very busy network causes a madly whirling string with a characteristic noise; a quiet network causes only a small twitch every few seconds. Placed in an unused corner of a hallway, the long string is visible and audible from many offices without being obtrusive. It is fun and useful. The Dangling String meets a key challenge in technology design for the next decade: how to create calm technology.

A collection of Mark’s essays, papers and presentations about Ubicomp are available on this website. Separately there are many research papers available online from Ubicomp conferences, e.g. Ubicomp 2008, Pervasive 2008.

So where are we now? How has the field progressed since Weiser first coined the term Ubiquitous Computing in 1988?

A very good critique paper is Yesterday’s tomorrows: notes on ubiquitous computing’s dominant vision by Genevieve Bell and Paul Dourish. In that paper they outline some of the failings and opportunities due to the massive influence Mark’s original vision had on Ubicomp. I particularly like their observations that in many ways we are already living in a Ubicomp world – technology and our lifestyles have merged over the last decade. Also of interest is their observation that Ubicomp environments are inherently messy.

Many many years ago myself and friends went through a short phase of playing Tic-Tac-Toe (TTT), also called Noughts and Crosses. You win this simple two player game by forming a horizontal, vertical or diagonal straight line of three Xs or three Os. If both players play properly then neither player can win and the game finishes as a draw.

Early in the 90s Jiajie Zhang, as part of his PhD, did a number of experiments where he tested how well people could play TTT against a computer. What he wondered was whether different visual representations effected people’s ability to play the simple childhood game of TTT. Below are four (A to D) different graphical and conceptual layouts of TTT. If you play the (C) Shape version of TTT would you win more or less than if you play against the computer on the (D) Colour version of TTT?

Each of these graphical representations is called an isomorph of TTT. When you use these isomorphs to play TTT you are trying to solve the same problem as in TTT except the problem is presented differently, i.e. to win you still need to create a group of three items.

When playing (A) Line you colour three circles in a line to win, (B) Number you colour three circles so the numbers add to 15, (C) Shape you colour three circles so they share at least one shape in common and (D) Colour you colour three circles so they share at least one common colour.

What he found was that people’s ability to play TTT, this simple childhood game, was seriously effected by what isomorph version of TTT they played on! In one experiment he found that more than 50% of the players failed to get 10 draws in a row in the first 50 games when they played on the (B) Number version.

Depending on the computer’s opening moves (he did a number of experiments where he controlled what sequence of opening moves the computer could make) he showed that (A) Line was easier than (D) Colour, which was easier than (C) Shape which was around the same hardness as (B) Number.

Not only did the visual representation effect how well players did, but what options they had for their next move on the visual representation also effected their playing ability.

For the full paper with far more details about the experimental results you should download at least one of Zhang’s papers about the research The Interaction between Perceptual and Cognitive Processes in a Distributed Problem Solving Task.

A good starting point for learning more about this area is to read about representational determinism and distributed cognition. Mike Scaife’s and Yvonne Rogers paper External cognition: how do graphical representations work? is a good introduction.

Do you think people’s ability to play the isomorphic versions of TTT was affected by their familiarity with the graphical layouts? If a player was given enough practice would they play TTT as well no matter how the game looked? If they could practice would the speed at which they learn to play better be due to the visual representation? Why would one visual representation be easier to learn with than another?

The benefit of this research is it highlights an interesting danger of an individual centered user experience: People could end up shaping and creating tools that make it subtly hard to use the tools for thinking and problem solving. For example if you were an accountant or scientist looking at lots of numbers there is software that makes it easier to create visualisations of the numbers: How do you know that your visualisation isn’t playing with your mind by making it harder to think about the numbers?

In today’s Creativity Knowledge post I’m focusing on a series of research projects from Patrick Baudisch, who is affiliated with the VIBE group in Microsoft Research and DUB in the University of Washington. Over the years Patrick and co-researchers have developed a range of innovative interaction techniques focused on enhancing people’s ability to move content around screens.

Interactive displays surfaces are getting larger – for example checkout these exciting demos of Multi-Touch and Surface (thanks to Cormac and co for the link). A usability problem is that content, such as icons, is moved around these large display surfaces using the same techniques as on desktops. We grab an icon with the mouse, we drag the icon by moving the mouse a physical distance proportional to on-screen distance, we then place the icon where we want it to go by letting go of the icon.

The desktop interaction model, which is designed for small displays, has been kludged to work on large surfaces. Imagine moving the icon with a large wall sized display: we grab the icon, we run half way across the room to drag the icon to a desired location, we let go of the icon, we catch our breath then run half way back across the room to where we started.

How would you make it easier to move content around large displays? How about when you can interact with the displays using your whole body?

Check out this interactive online flash demo where you can play with drag-and-pop. Instructions for the demo are here. Drag-and-pop is designed so when a “user starts dragging an icon towards some target icon, drag-and-pop responds by temporarily moving potential target icons towards the user’s current cursor location, thereby allowing the user to interact with these icons using comparably small hand movements”. On the drag-and-pop website you can find videos and papers explaining the interaction technique in detail, along with material explaining push-and-pop, drag-and-pick and Mountaz Hascoët’s push-and-throw. Yep all this pushing, dragging and popping is confusing! Enjoy.

On a related point: I’m regularly drawn to research around innovative interaction techniques but is there too much focus on novel interactions in interaction design research? Are we like moths to the flame of novelty? Does novelty distract from the development of deeper theories? Or does it highlight that HCI can and should also be practiced as an applied field of research?

Look at Smash Magazine’s recent article on Modern Approaches To Data Visualization for a quick overview of key information visualizations.

Admire the 500+ visualizations at visualcomplexity.

Create your own online visualizations at Many Eyes.

Today I’m adding a new category called Creativity Knowledge to the User Designer blog. Creativity Knowledge will feature short posts highlighting specific examples of Human Computer Interaction and Interaction Design research. Posts in Creativity Knowledge will help flesh out (often very) distantly related research around enabling anyone to take control and shape their own user experiences.

Why am I adding Creativity Knowledge? Around 10+ years ago when I was getting into HCI and Interaction Design I remember finding lots of academic papers that dryly outlined exciting-vibrate-in-your-chair ideas and design concepts. Many of the prototyped ideas had conceptual and research histories dating back decades. Some of those ideas made it into products, such as the iPhone, others have been forgotten, gotten lost among a sea of publications and projects, or were before their time.

Not being aware of what’s out there is depriving yourself of a powerful source of inspiration, because you don’t know how different ideas could weave together. Maybe some of the research will inspire you to create or enhance a product, or spin out a startup?

Best of all it can be lots of fun seeing what others have invented!