When we communicate face-to-face, being able to see the person we're talking to makes it easier to understand speech, especially when the acoustic signal is degraded by noise or hearing loss. We call these benefits
audiovisual speech enhancement.
Development of audiovisual speech enhancement in children
Visual speech can help us to know when to listen, supplement masked auditory speech information, and help to separate speech from similar competing sounds. Our first project is investigating how well children at various ages can use visual speech in these different ways. Experiments examine how sensitive children are to different audiovisual cues and how much these different mechanisms contribute to individual differences in children's audiovisual speech enhancement. This project's long term goals are to provide a cohesive theoretical account of the development of AV speech enhancement, and ultimately to improve AV communication outcomes for children with hearing loss. This project is funded by a NIH Centers for Biomedical Research Excellence (COGRE) grant (NIH-NIGMS / 5P20GM109023-04).
Effects of hearing aid compression on temporal cues in audiovisual speech
Our second project is investigating how wide dynamic range compression (WDRC)—a key feature in hearing aids—affects the timing cues in audiovisual speech, and consequently affect listeners' ability to benefit from visual speech cues. Hearing aids and visual speech (lip reading) are two strategies recommended by audiologists to enhance speech understanding in noise. Mouth movements help listeners to track the temporal amplitude envelope of auditory speech (the slow time-varying changes in signal energy). This help listeners direct auditory analyses to the speech signal of interest, rather than surrounding background noise. Hearing aids with WDRC amplify low intensity parts of auditory signals more than the higher-intensity parts, in order to make more of the signal audible. This distorts the temporal amplitude envelope of auditory speech.  Distortions of the auditory amplitude envelope likely disrupt the natural correspondence between mouth movements and the temporal amplitude envelope of auditory speech. Current experiments examine which timing cues are important for audiovisual speech enhancement, how WDRC affects the correspondence between mouth movements and the temporal amplitude envelope of auditory speech, and whether WDRC affects audiovisual speech enhancement in individuals with normal hearing. The long-term goal of this project is to understand the impact of WDRC on audiovisual speech perception in individuals with hearing loss. This project is funded by the Greater Plains IDeA Clinical and Translation Research Grant, through the National Institute of General Medical Sciences, 1U54GM115458-01.