CNL Lunch

Virginie van Wassenhove

An Event-Related Potential Study of Auditory-Visual Speech

Thursday November 14th 2002, 12:30pm, 3416 Marie Mount Hall

This talk will provide new and surprising results of electroencephalographic recordings of auditory-visual speech syllables. General principles of multisensory integration (Stein and Meredith, 1983) and more recent fMRI recordings (Calvert, 1997) have suggested that presentation of multisensory events lead to an enhanced neural response of polymodal (Superior Temporal Sulcus) and unimodal sensory cortices (auditory cortex). Furthermore, recent EEG studies have focused on non-speech stimuli and found early signal amplitude enhancement to the presentation of AV stimuli (Giard and Peronnet, 1999). Based upon those general findings and evidence that lip-reading benefits auditory speech perception, we predicted an amplitude enhancement of speech related ERPs for auditory-visual speech as compared to auditory alone stimuli.

Our results show the opposite effect: the amplitude of the auditory N1/P2 complex was reduced in both congruent (AV syllables /ka/, /pa/, /ta/) and incongruent (McGurk fusion) AV speech stimuli. The recorded ERPs do not provide a straightforward interpretation of the behavioral facilitation effects characteristic of multimodal integration and auditory-visual speech perception. Overall, the electrophysiological trend suggests that visual kinematics engage pre-phonetically and pre-attentively in speech perception. The amplitude reduction indicates that auditory cortices may be constrained to extract features of speech unavailable or ambiguous (VOT) in the visual domain. Most importantly, the 200ms post-stimulus range appears to carry crucial information about the phonological representativeness of a syllable, affecting subsequent processing such as sublexical access up to about 350ms