mlsanityData to 14 Oct 2026

Homophone Disambiguation Reveals Patterns of Context Mixing in Speech Transformers

arXiv 2310.09925 ↗First posted 15 Oct 2023cs.CL cs.AI cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Transformers have become a key architecture in speech processing, but our understanding of how they build up representations of acoustic and linguistic structure is limited. In this study, we address this gap by investigating how measures of 'context-mixing' developed for text models can be adapted and applied to models of spoken language. We identify a linguistic phenomenon that is ideal for such a case study: homophony in French (e.g. livre vs livres), where a speech recognition model has to attend to syntactic cues such as determiners and pronouns in order to disambiguate spoken words with identical pronunciations and transcribe them while respecting grammatical agreement. We perform a series of controlled experiments and probing analyses on Transformer-based speech models. Our findings reveal that representations in encoder-only models effectively incorporate these cues to identify the correct transcription, whereas encoders in encoder-decoder models mainly relegate the task of capturing contextual dependencies to decoder modules.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 27 of its 34 references that are in the map. First posted on arXiv 15 Oct 2023.

Builds on

27 of 34 references.

References · 27

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  6. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  7. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  8. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 501
  9. 2019Revealing the Dark Secrets of BERTcited by 301
  10. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 298
  11. 2020Explaining by Removing: A Unified Framework for Model Explanationcited by 218
  12. 2019Visualizing and Measuring the Geometry of BERTcited by 216
15 more
  1. 2019Open Sesame: Getting Inside BERT's Linguistic Knowledgecited by 120
  2. 2022Measuring the Mixing of Contextual Information in the Transformercited by 65
  3. 2021What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structurecited by 52
  4. 2021Incorporating Residual and Normalization Layers into Analysis of Masked Language Modelscited by 51
  5. 2021LeBenchmark: A Reproducible Framework for Assessing Self-Supervised Representation Learning from Speechcited by 47
  6. 2022GlobEnc: Quantifying Global Token Attribution by Incorporating the Whole Encoder Layer in Transformerscited by 44
  7. 2023Quantifying Context Mixing in Transformerscited by 43
  8. 2020Unsupervised Pre-training of Bidirectional Speech Encoders via Masked Reconstructioncited by 38
  9. 2019Improving Transformer-based Speech Recognition Using Unsupervised Pre-trainingcited by 30
  10. 2022Do self-supervised speech models develop human-like perception biases?cited by 28
  11. 2022How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformerscited by 21
  12. 2023Wave to Syntax: Probing spoken language models for syntaxcited by 18
  13. 2023Putting Natural in Natural Language Processingcited by 17
  14. 2023Investigating Pre-trained Audio Encoders in the Low-Resource Conditioncited by 11
  15. 2022Analysis of Self-Attention Head Diversity for Conformer-based Automatic Speech Recognitioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 6; 2026: 2.

13 papers.

Sort:

By area:

  1. 2024Self-Supervised Speech Representations are More Phonetic than Semanticcited by 71
  2. 2024Twists, Humps, and Pebbles: Multilingual Speech Recognition Models Exhibit Gender Performance Gapscited by 25
  3. 2024Mechanistic?cited by 25
  4. 2024Human-like Linguistic Biases in Neural Speech Models: Phonetic Categorization and Phonotactic Constraints in Wav2Vec2.0cited by 19
  5. 2025Are We Paying Attention to Her? Investigating Gender Disambiguation and Attention in Machine Translationcited by 7
  6. 2025Layer-wise Minimal Pair Probing Reveals Contextual Grammatical-Conceptual Hierarchy in Speech Representationscited by 5
  7. 2025Voice, Bias, and Coreference: An Interpretability Study of Gender in Speech Translationcited by 5
  8. 2025Trick or Neat: Adversarial Ambiguity and Language Model Evaluationcited by 3
  9. 2025The Unheard Alternative: Contrastive Explanations for Speech-to-Text Modelscited by 3
  10. 2024How Language Models Prioritize Contextual Grammatical Cues?cited by 1
  11. 2025Cross-Attention is Half Explanation in Speech-to-Text Modelscited by 1
  12. 2026Layer-wise Probing of wav2vec 2.0 and Whisper for Consonant Cluster Reduction in African American Englishcited by 0
1 more
  1. 2026HearInContext: A Benchmark for Implicit Context in Speech Recognitioncited by 0