Linda Drijvers | The neural mechanisms of how iconic gestures boost degraded speech comprehension in native and non-native listeners

During face-to-face communication, listeners often integrate auditory input, such as speech, with visual input, such as iconic co-speech gestures. These iconic gestures can enhance speech comprehension in adverse listening conditions. In this talk, I will address how the neural integration of iconic gestures and speech is modulated by speech degradation and native listener status, and how this can be indexed by event-related potentials, such as the N400. Furthermore, I will address how modulations of low- and high-frequency oscillatory power in language, motor & visual areas of the brain support gestural enhancement of degraded speech comprehension in native and non-native listeners. I will present a series of experiments where we propose a mechanistic role for oscillatory brain dynamics in engaging brain areas that contribute to multimodal semantic integration. In these studies, we demonstrated that low- and high-frequency oscillations with distinct spatiotemporal characteristics can predict the degree of integration of audiovisual information in a semantic context. These integration processes distinctly differ for native and non-native listeners, and suggest different processing strategies during multimodal speech comprehension in adverse listening conditions.

Poster