Picture the following, and prepare to be amazed. You're walking across a college campus when a stranger asks you for directions. While you're talking to him, two men pass between you carrying a wooden door. You feel a moment's irritation, but they move on and you carry on describing the route. When you've finished, the stranger informs you that you've just taken part in a psychology experiment. "Did you notice anything change after the two men passed with the door?" he asks. "No," you reply uneasily. He then explains that the man who initially approached you walked off behind the door, leaving him in his place. The first man now comes up to join you. Looking at them standing side by side, you notice that the two are of different height and build, are dressed differently, have different haircuts and different voices.

It sounds impossible, but when Daniel Simons, a psychologist at Harvard University, and his colleague Daniel Levin of Kent State University in Ohio actually did this experiment, they found that fully 50 per cent of those who took part failed to notice the substitution. The subjects had succumbed to what is called change blindness. Taken with a glut of recent experimental results, this phenomenon suggests we see far less than we think we do.

Rather than logging every detail of the visual scene, says Simons, we are actually highly selective about what we take in. Our impression of seeing everything is just that-an impression. In fact we extract a few details and rely on memory, or perhaps even our imagination, for the rest. Others have a more radical interpretation: they say that we see nothing at all, and our belief that we have only to open our eyes to take in the entire visible world is mistaken-an illusion.

Until the last decade, vision researchers thought that seeing really meant making pictures in the brain. By building detailed internal representations of the world, and comparing them over time, we would be able to pick out anything that changed. Then in 1991, in his book Consciousness Explained, the philosopher Daniel Dennett made the then controversial claim that our brains hold only a few salient details about the world-and that this is the reason we are able to function at all.

We don't store elaborate pictures in short-term memory, Dennett said, because it isn't necessary and would take up valuable computing power. Rather, we log what has changed and assume the rest has stayed the same. Of course, this is bound to mean that we miss a few details. Experimenters had already shown that we may ignore items in the visual field if they appear not to be significant-a repeated word or line on a page of text, for instance. But nobody, not even Dennett, realised quite how little we really do "see".

Just a year later, at a conference on perception in Vancouver, British Columbia, John Grimes of the University of Illinois caused a stir when he described how people shown computer-generated pictures of natural scenes were blind to changes that were made during an eye movement. Dennett was delighted. "I wish in retrospect that I'd been more daring, since the effects are stronger than I claimed," he says. Since then, more and more examples have been found that show just how illusory our visual world is. It turns out that your eyes don't need to be moving to be fooled. In a typical lab demonstration, you might be shown a picture on a computer screen of, say, a couple dining on a terrace. The picture would disappear, to be replaced for a fraction of a second by a blank screen, before reappearing significantly altered-by the raising of a railing behind the couple, perhaps. The picture flickers back and forth, and many people search the screen for up to a minute before they see the change. A few never spot it.

It's an unnerving experience. But to some extent "change blindness" is artificial because the change is masked in some way. In real life, there tends to be a visible movement that signals the change. But not always. As Simons points out, "We have all had the experience of not noticing a traffic signal change because we had briefly looked away." And there's a related phenomenon called inattentional blindness, that doesn't need any visual trick at all: if you are not paying attention to some feature of a scene, you won't see it. In our own simple demonstration, few people spot that the first "t" in "New Scientist" is not the same on the cover as on page 27.

Last year, with Christopher Chabris, also at Harvard, Simons showed people a videotape of a basketball game and asked them to count the passes made by one or other team. After about 45 seconds, a man dressed in a gorilla suit walked slowly across the scene, passing between the players. Although he was visible for five seconds, 40 per cent of the viewers failed to notice him. When the tape was played again, and they were asked simply to watch it, they saw him easily. Not surprisingly, some found it hard to believe it was the same tape.

Now imagine that the task absorbing their attention had been driving a car, and the gorilla-man had been a pedestrian crossing their path. According to some estimates, nearly half of all fatal motor-vehicle accidents in the US can be attributed to driver error, including lapses in attention. It is more than just academic interest that has made both forms of cognitive error hot research topics. Such errors raise important questions about vision. For instance, how can we reconcile these gross lapses with our subjective experience of having continuous access to a rich visual scene? Last year, Stephen Kosslyn of Harvard University showed that imagining a scene activates parts of the visual cortex in the same way as seeing it. He says that this supports the idea that we take in just what information we consider important at the time, and fill in the gaps where the details are less important. "The illusion that we see 'everything' is partly a result of filling in the gaps using memory," he says. "Such memories can be created based on beliefs and expectations."

Ronald Rensink of the University of British Columbia in Vancouver believes that our impression of a rich visual world comes from our building internal representations, though he accepts that they are far less detailed than was once thought. According to his "coherence theory", the brain first constructs a temporary layout of the visual scene-not much more than the basic geometry and light distribution. Then attention comes along and pulls out a few of these "proto-objects" to a higher resolution. More importantly, he explains, "what attention does is to stabilise these representations so that they form an individual object, something with continuity in space and in time". The moment attention is released, they dissolve back into the volatile, unresolved landscape. In Rensink's view, focused attention is needed to perceive change.

But while Rensink or Kosslyn would argue that there is some role for internal images or memory, other researchers argue that we can get the impression of visual richness without holding any of that richness in our heads. Back in 1992, Kevin O'Regan, an experimental psychologist at the French National Centre for Scientific Research (CNRS) in Paris put forward what later became known as his "grand illusion" theory. He argued that we hold no picture of the visual world in our brains. Instead, we refer back to the external visual world as different aspects become important. The illusion arises from the fact that as soon as you ask yourself "am I seeing this or that?" you turn your attention to it and see it.

According to O'Regan, it's not just our impression of richness that is illusory, but also the sense of having control over what we see. "We have the illusion that when something flickers outside the window, we notice it flickering and decide to move our eyes and look," says Susan Blackmore of the University of the West of England, who supports O'Regan's views. "That's balderdash." In fact, she says, we are at the mercy of our change detection mechanisms, which automatically drag our attention here, there and everywhere.

At a meeting in Brussels in July this year, O'Regan and Alva No' of the University of California, Santa Cruz, updated the controversial theory. Sensation, whether it be visual, auditory or tactile, is not something that takes place in the brain, they argue. Rather it exists in the knowledge that if you were to perform a certain action, it would produce a certain change in sensory input. "Sensation is not something that we feel, but sensation is something that we do," says O'Regan.

According to this idea, the sensation of "redness" arises from knowing that moving your eyes onto a red patch will produce a certain change in the pattern of stimulation in line with laws of redness. In other words, the role of the brain is to initiate the exploratory action and to hold the knowledge of those laws: together this give rise to the sensation of redness.

Once you dismiss the need for visual memory, O'Regan says, many of the problems that vision researchers have grappled with for decades vanish. Namely, how does the ropy physics of the eye give rise to the largely flawless experience of visual perception? Leaving aside the blind spot in each retina and the fact that we view the world through a jerky sequence of eye movements or "saccades", we have two upside-down retinal images. If you assume that our brains build detailed reconstructions from such inadequate, distorted input, you have to postulate some kind of compensation mechanism in the visual system. In O'Regan's model no such mechanism is required because there is no reconstruction. His theory also explains change blindness. Take the example of the dining couple. The reason you don't notice the raising of the railing is because you didn't notice the railing in the first place-it wasn't relevant so it remained invisible.

O'Regan's ideas have not been generally accepted. "He's pushed the idea that we lack visual representations farther than most people in the field have been willing to," says Simons. But despite their differences, Simons, Rensink and O'Regan all say that of all the myriad visual details of any scene that you could record, you take only what is relevant to you at the time.

In the Simons-Levin experiment, for example, even the object to which the person is attending-the stranger asking for directions-can be swapped without them noticing. Despite the fact that they were looking at him for around a minute, half the subjects encoded none of the details of his physical appearance that were later to change. It was not relevant that the stranger had a certain haircut or that his trousers were a certain colour. What was relevant was that he was a person in a certain location addressing them with a certain query. "Paying attention to an object does not give you all of that object's properties for free," says Simons. He points out that those who did notice the switch were students of about the same age as the "strangers". Being in the same social group, he and Levin speculated, they would be more inclined to take in individual details, whereas older subjects might categorise the stranger as "student" and leave it at that.

The relationships between attention, awareness and vision have yet to be clarified. But there is one thing on which most researchers agree: because we have a less than complete picture of the world at any one time, there is the potential for distortion and error. And that has all sorts of implications, not least for eyewitnesses. If it is possible to stand less than a metre from a person and talk to them for a minute without taking in more than a few basic facts, how reliable is the testimony of a person who witnesses a scene from a distance, when they were oblivious to its significance and only later came to recall it?

"In my view, imagery plays a key role in many sorts of false memories," says Kosslyn. "One is 'filling in' the gaps and later remembering not only what was attended to, but also what was filled in." In retrospect, he says, we don't make any distinction between the two types of information. For all our experience of a rich visual world, it seems that we take in no more than a handful of facts about the world, throw in a few stored images and beliefs, and produce a convincing whole in which it is impossible to tell what was real and what imagined. As Blackmore puts it: "There is a world and a brain in it, which together are building a construction, a story, a great confabulation."

Laura Spinney is a writer based in London

Further reading: "Failure to detect changes to people during real-world interaction" by Daniel J. Simons and Daniel T. Levin, Psychonomic Bulletin and Review, vol 4, p 644 (1998)
"Solving the 'real' mysteries of visual perception: the world as an outside memory" by J. K. O'Regan, Canadian Journal of Psychology, vol 46, p 461 (1992)
"Beyond the grand illusion: what change blindness really teaches us about vision" by A. No' and others, Visual Cognition, vol 7, p 93 (2000). To try the "dining couple" experiment and others, see http://nivea.psycho.univ-paris5.fr/ASSChtml/ASSC.html

New Scientist issue: 18 November 2000

PLEASE MENTION NEW SCIENTIST AS THE SOURCE OF THIS STORY AND, IF PUBLISHING ONLINE, PLEASE CARRY A HYPERLINK TO: http://www.newscientist.com