Why Is It Harder to Understand English When You Cannot See the Speaker?

Listening has never been purely auditory

Think about a noisy café.

Someone across the table says:

Did you say fifteen or fifty?

The sound reaching your ears may be imperfect. Music is playing. Cups are hitting tables. Another conversation overlaps with yours.

But you are not limited to the waveform.

You can see when the speaker begins talking. You can see the rhythm of the mouth. Some consonants have visible articulatory information. You can watch the speaker's face and follow a gesture that narrows the intended meaning.

None of these cues gives you a transcript of the sentence.

Together, however, they can reduce uncertainty.

Research on audiovisual speech has documented this visual contribution for decades. Reviews describe a robust visual gain in speech intelligibility when listeners can see a talker, particularly when the auditory signal is degraded (Peelle & Sommers, 2015; Irwin & DiBlasi, 2017).

The important word is combine.

Your brain does not necessarily finish listening and then inspect the face for a second opinion. Visual information can influence the speech percept itself.

Your mouth gives phonetic clues before the word is finished

Visible speech is sometimes casually called “lip-reading,” but that label can make the process sound more complete than it is.

Many speech sounds are visually ambiguous. Different sounds can produce similar-looking mouth shapes. Other articulatory differences happen largely inside the mouth and are difficult to see.

The face therefore does not replace the soundtrack.

It constrains it.

Suppose noisy audio leaves several words plausible. A visible lip closure can make some candidates more likely and others less likely. Timing from the mouth can also help the listener anticipate when acoustic information is about to arrive.

This is a probabilistic advantage rather than a visual transcription system.

That distinction matters for language learners. You do not need to become an expert lip-reader to benefit from seeing a speaker.

But extremely bad audio can break the partnership

There is another limit that sounds paradoxical.

If visual information helps when sound gets worse, should it help most when the sound is almost unintelligible?

Not necessarily.

Linda Drijvers and Aslı Özyürek tested highly proficient non-native listeners with clear, moderately degraded and severely degraded English speech. Participants saw blurred lips, visible speech, or visible speech plus an iconic gesture.

The non-native listeners benefited from visual information, but less than native listeners in the comparable work. More importantly, visible-speech benefit was minimal for non-native listeners when the auditory signal was severely degraded. The authors argue that L2 listeners may need enough phonological information in the sound to integrate the visual cues effectively (Drijvers & Özyürek, 2020).

So audiovisual benefit can have a useful middle zone.

When audio is clear, you may not need much rescue.

When it is moderately difficult, the face can disambiguate.

When it becomes extremely poor, there may not be enough auditory structure left for an L2 listener to connect the streams efficiently.

That is far more interesting than “watch the lips.”

Gestures can tell you about meaning, not just sound

Imagine someone says a verb you almost recognize while making a clear twisting motion with one hand.

The gesture does not show you the consonants.

It may still make the intended action easier to infer.

In a study of native and highly proficient non-native English listeners, iconic gestures and visible speech helped listeners understand degraded speech, although the non-native group generally gained less from the visual information (Drijvers & Özyürek, 2020).

An earlier L2 listening study by Sueyoshi and Hardison used a videotaped English lecture with low-intermediate and advanced learners. Both proficiency groups scored better when visual cues were available than in the audio-only condition, but the pattern differed by proficiency: the advanced group performed best with the face condition, while the lower-proficiency group performed best when gesture and face were both available (Sueyoshi & Hardison, 2005).

That does not establish a universal rule that beginners “need gestures” and advanced learners “need faces.”

It does show why visual support is not one thing.

A mouth can narrow a sound.

A gesture can narrow a meaning.

Why second-language listeners may use the face differently

Visual speech is not a universal code that every learner reads identically.

Your language experience changes which sound distinctions matter to you and how much value you assign to auditory and visual cues.

Debra Hardison's work with advanced learners from Japanese, Korean, Spanish and Malay backgrounds found that matched visual cues could improve identification of particular English consonants and that first-language experience influenced audiovisual integration (Hardison, 1996).

Other research has found that the native-language backgrounds of both listener and speaker can alter the amount and pattern of audiovisual benefit in noise (Yi et al., 2014).

This means two learners can watch exactly the same English speaker and extract different amounts of useful information from the face.

The visual signal is the same.

The perceptual history interpreting it is not.

Seeing a face is not guaranteed to improve comprehension

At this point it would be easy to turn the research into a new rule:

Always study English with video.

The evidence is not that clean.

A study by Nobuhiro Kamiya compared L2 English listening tasks in three conditions: upper-body video, close-up face video and audio only, across easier and harder materials. The effects of modality were limited rather than uniformly favouring one visual condition (Kamiya, 2025).

Different experiments also measure different outcomes: identifying a word in noise is not the same task as understanding a lecture, remembering a story or learning vocabulary.

And visual information can demand attention as well as provide information.

The correct conclusion is conditional:

Seeing the speaker can make speech easier to identify when the visible cues are informative and the listener can integrate them with the available audio.

That is less catchy than “video beats audio.”

It is also much more useful.

What this changes about listening practice

You do not need to choose between “video learner” and “audio learner.”

Instead, change the available information depending on what you are training.

If a difficult video becomes understandable when you can see the speaker, pay attention to why. Was the mouth helping you identify words? Did gesture reveal meaning? Was the visual scene simply giving you context?

Then remove one support.

Listen to the same or comparable speech without the face. If comprehension collapses, you have discovered something specific about your auditory recognition rather than proving that the earlier comprehension was fake.

The practical loop can be:

1. Use natural audiovisual speech to experience the full signal.
2. Try audio only to expose what your ears cannot yet resolve.
3. Return to the video and locate the visual clue that changes the percept.
4. Listen again without video and see whether the sound has become identifiable.

That fits naturally with the broader listen → check → listen approach. The difference is that the “check” is not always text. Sometimes the missing evidence is on the speaker's face.