Ockham's Razor

As a young child Professor John Bradshaw from the Department of Psychology at Monash University, was fascinated by the Punch and Judy puppet plays and the very convincing ventriloquistic effects. This later led him to research how we hear - is it the left side of the brain or the right side that dominates our auditory signals? And do mouth movement asymmetries impact on our ability to lip read?

Transcript

This transcript was typed from a recording of the program. The ABC cannot guarantee its complete accuracy because of the possibility of mishearing and occasional difficulty in identifying speakers.

Robyn Williams: What is so very creepy about a ventriloquist's dummy? Is it the fixed smile? Is it the half-living, half-dead aura, or the way the strangely dapper doll insists on insulting the chap whose knee he sits on" Is it the way he has to keep saying 'Gottle of Gear, Gottle of Gear' (instead of Bottle of Beer), or is it the way the dummy comes to life in old movies and goes to strangle his master?

Of course there is a psychological element, and that's what John Bradshaw is addressing today, although I must admit a certain surprise when he said he wanted to take Ockham's Razor to ventriloquism. But there's always a first time, isn't there? John Bradshaw is Professor of Neuropsychology at Monash University.

John Bradshaw: As a very young child towards the end of World War II, I was often taken for summer holidays to Llandudno, a staid 'watering place', as such resorts were then commonly known, to distinguish them from the much more raunchy holiday venues like Blackpool. On the 'Welsh Riviera' - yet another euphemism - it is noted for healthy and bracing air (which meant that it was often cold, wet and windy) and for its imposing Victorian hotels with palm courts, where the orchestra played while you partook of genteel afternoon tea. Forbiddingly large, impassive and unwelcoming landladies ruled the cheaper end of town, with boarding establishment boasting inappropriately optimistic nameplates, such as Costa Bella, Mira Mar, Ozone House and, of course, Sea View. Redolent with the ubiquitous stale smells of overcooked meat and the obligatory three veg, they were almost an enticement in a period of general wartime privation.

On the pier beside the Dodgem Cars stood the Punch and Judy show. Children loved it, before TV came, and I could never get enough of the knockabout puppet play, with its fascinating and convincing ventriloquistic effects, whereby the characters continually abused each other. How did the puppeteer make the speech appear to originate so naturally and appropriately from the wooden mouth of each little doll, the right one in the right place at the right time?

Some 20 or so years ago I was faced with a similar research problem. But first, let me take you back again to the closing days of World Wear II. It was the beginning of automated warfare, when equipment operators were faced with the problem of too many things happening too fast all at once, more than could be coped with. Nowadays we call it Information Overload, but a young researcher, Donald Broadbent, later a Fellow of the Royal Society, was studying our 'channel capacity', or ability to follow simultaneously more than one message through earphones. Apart from the obvious relevance of such applied research, the studies demonstrated much that was not then known about attentional holding, switching, focusing and sharing. They also showed that when two messages went simultaneously, via earphones, separately one to each ear, the message content in the right ear tended to dominate, though other features such as emotional aspects or personal identity, may be better identified from the left ear. All this agreed with neurological observations from the period's many war-wounded; the brain's left side or hemisphere, to which the right ear preferentially sends its input, is where language tends to be elaborated.

From the 1950s until the 1980s it was held that Broadbent's 'Ear Effects', as they came to be known, depended upon the belief that right ear inputs, in addition to projecting more strongly to the opposite left half brain also suppressed the left ear's weaker, uncrossed signal, direct to that same destination on the left. Similarly, the left ear input, in addition to projecting strongly to the right hemisphere, also suppressed the right ear's weaker, uncrossed signal to that same side. This account also seemingly explained why ear asymmetries were at best very weak without competition between ears.

In the 1980s, with my student Jane Pierson, I wondered whether such ear asymmetries really needed pathway competition between the ears; could it instead be a matter of attentional competition? We first showed that ear asymmetries were in fact even more striking if both signals went to a single earphone, the messages being spoken by a male and a female voice to facilitate channel separation; the right ear clearly dominated the left in this purely monaural language-related task, as long as the two messages competed for attention.

However a clincher for an attentional explanation would be the demonstration of an apparent right ear superiority when both signals in fact went equally to both ears, from directly in front and behind, but when ventriloquistic deception made it appear that they instead came from one side, right. Note, first, that the location of a sound source, which lies directly in front of or behind a listener, is systematically ambiguous. We automatically judge direction of origin of a sound partly by very slight differences of asynchronies, in times of arrival of a signal to the two ears. There is no such asynchrony with sound emitters lying on extensions, fore or aft, of our midlines, and our ability to localise improves as the sound emitter moves to an ever-more-lateral position.

So, we hid the loudspeakers, one directly in front of and the other behind the listener, who was led to believe that the signals instead came from a third, highly visible, speaker on the left or right, which continually emitted buzzes, hums, crackles and pops, thereby ventriloquistically 'capturing' the apparent location of the male and female voices, just as in a puppet show! As before, the listener tried to follow and report on one of the two voices, and performed far better when they seemed to come from the right. The ventriloquistic deception thus proved extraordinarily compelling, indicating that Broadbent's ear asymmetries were attentional, rather than anatomical in origin, depending little or not at all upon pathway suppression, or where listeners actually received the message (left or right ear), and more on attentional processes and where they thought the message came from.

Visual capture of an auditory channel, the basis of ventriloquism, was demonstrated in a different way in the 1970s by Harry McGurk, in studies on speech recognition. The eponymously named McGurk illusion arises when speech sounds and lip movements are incongruent, failing to match, as in a badly-dubbed movie. Thus, if on a videotape the sound 'ba' is dubbed over the mouth movements of a speaker saying 'Gavin Attwood:', listeners tend to report hearing 'da', which is exactly intermediate between the two, in terms of where and how in a speaker's mouth the speech sounds would be made. We therefore see that visual processing affects auditory experience in normal listeners, another form of visual capture.

But let me first digress again and note some other interesting and relevant neurological phenomena manifested by normal healthy individuals. The left side of the face, controlled by the right or 'emotional' half brain, tends to express emotions more strongly than the right side of the face. In contrast, the right side of the face, particularly the mouth, seems more important in the expression of speech, as shown by measurements of photographs of speakers' mouth and lip movements; this of course again reflects control of the speech apparatus, predominantly by the left half brain. Watch a newsreader on television: eight out of ten show noticeably more extensive speech-related mouth movements or gestures by the mouth or lips on the speaker's right hand side. (Remember, the speaker's right side is from the observer's viewpoint on the left side of the screen.

So, might mouth movement asymmetries also impact on our ability to lip read? Here, with my friend and colleague, Mike Nicholls at Melbourne University, we employed the McGurk effect. With an electronic mask we exactly covered the left or right halves of the mouth in video recordings of the full face of the speakers, or as a control left both sides uncovered. As expected, with incongruent visual and auditory signals, the speakers apparently saying one thing at a visual level, and something different at an auditory level, we found McGurk-type errors; people's reports of what they thought they had heard were influenced by what they had simultaneously seen. These errors were particularly prominent when both sides of the mouth were clearly visible in the control condition. The McGurk effect however, was considerably attenuated when the right side of the mouth (from the speaker's perspective) was electronically masked, demonstrating that this side is more important to lip reading than is the mouth's left side. Again, it is presumably the more expressive side during normal speech. The fact that the McGurk effect was no stronger when both sides of the mouth were visible, than when the left side was covered and the right side visible, suggests that the visual information provided by movements of the right side of the mouth is just as informative as that provided by movements of the full, unmasked mouth. So in principle, we can lip read from someone with only half a face, as long as the unfortunate individual's mutilation spares the right side.

We also mirror-reversed the images to test the possibility that such asymmetries may additionally be affected by an observer-bias towards the left side, as seen by the observer, of images like the face. We do know that we attend somewhat more to the left sides of faces, seen in full frontal view, from studies employing so-called chimeric or composite faces. Photographs of individuals smiling or frowning are cut down the midline, and recombined so that the smile is on the left and the frown on the right, or vice versa. People tend to judge such composites as appearing overall more or less happy, or unhappy, on the basis of whether the particular emotion is represented on the left side, as viewed. The emotionally is generally more strongly developed on the face's actual, physical, left side, that is, from the perspective of the owner of the face, due to that side's direct access to emotional centres in the right half brain.

However, mirrorreversing the composites shows that a leftwards perceptual bias is largely independent of any motor differences in generating the expression. A leftwards bias, of course, directs attention towards the more expressive right side of a speaker's face, where lip reading may receive a boost. Is this one evolutionary reason why we tend to look slightly leftwards at a speaker during conversation? Or is it just a by-product of the perceiver's own emotional arousal, whereby activation of emotional centres in the right half brain 'overflows', to cause a slight leftward shift in the focus of attention?

I wonder whether children watching a puppet show - though I now fear that Punch and Judy has followed the dinosaurs and the Tasmanian Tiger into extinction - tend to look more towards the left when watching the puppets' faces? We do of course still see 'speaking' puppets on children's TV; maybe there is a lesson to be learnt in trying to make as congruent as possible the relationships between the seen mouth movements and the soundtrack, to avoid unwanted McGurk effects. And the puppet masters should perhaps play particular attention to the right sides of the charges' faces, to achieve maximum verisimilitude. Isn't that what ventriloquism is all about? And if you find totally confusing all this left and right business - brain, side of body, and side of a viewed image, whether or not mirror reversed - please don't worry! Even after many years I still do too!

Robyn Williams: So, split brains again. Last week it was links to schizophrenia and split brains; this week, a problem of perception. Anyway, if you want to drawl 'Make my day' out of the side of your mouth like Clint Eastwood, make sure it's the right hand side.

John Bradshaw is Professor of Neuropsychology at Monash University.

Next week, a challenge to global warming orthodoxies: Professor Don Aitkin, a former Vice Chancellor at the University of Canberra, says why we may have got the warnings wrong.

I'm Robyn Williams.

Guests

Dr John Bradshaw
Professor of Neuropsychology
Monash University
Melbourne

Presenter

Robyn Williams

Producer

Brigitte Seega