mlsanityData to 14 Oct 2026

The Unheard Alternative: Contrastive Explanations for Speech-to-Text Models

arXiv 2509.26543 ↗First posted 30 Sep 2025cs.CL cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Contrastive explanations, which indicate why an AI system produced one output (the target) instead of another (the foil), are widely regarded in explainable AI as more informative and interpretable than standard explanations. However, obtaining such explanations for speech-to-text (S2T) generative models remains an open challenge. Drawing from feature attribution techniques, we propose the first method to obtain contrastive explanations in S2T by analyzing how parts of the input spectrogram influence the choice between alternative outputs. Through a case study on gender assignment in speech translation, we show that our method accurately identifies the audio features that drive the selection of one gender over another. By extending the scope of contrastive explanations to S2T, our work provides a foundation for better understanding S2T models.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 38 of its 69 references that are in the map. First posted on arXiv 30 Sep 2025.

Builds on

38 of 69 references.

References · 38

  1. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  2. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  3. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,206
  4. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 599
  5. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  6. 2015Evaluating the visualization of what a Deep Neural Network has learnedcited by 419
  7. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 226
  8. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 176
  9. 2019Sanity Checks for Saliency Metricscited by 119
  10. 2020If beam search is the answer, what was the question?cited by 116
  11. 2019Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debatescited by 114
  12. 2022Interpreting Language Models with Contrastive Explanationscited by 92
26 more
  1. 2017Explaining Recurrent Neural Network Predictions in Sentiment Analysiscited by 88
  2. 2023Inseq: An Interpretability Toolkit for Sequence Generation Modelscited by 78
  3. 2021Contrastive Explanations for Model Interpretabilitycited by 75
  4. 2020CoVoST 2 and Massively Multilingual Speech-to-Text Translationcited by 74
  5. 2020Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpuscited by 57
  6. 2023Post Hoc Explanations of Language Models Can Improve Language Modelscited by 50
  7. 2022Towards Opening the Black Box of Neural Machine Translation: Source and Target Interpretations of the Transformercited by 45
  8. 2023Explaining How Transformers Use Context to Build Predictionscited by 43
  9. 2023Sparks of Large Audio Models: A Survey and Outlookcited by 41
  10. 2024How to Compute the Probability of a Wordcited by 36
  11. 2021Rationales for Sequential Predictionscited by 26
  12. 2022Under the Morphosyntactic Lens: A Multifaceted Evaluation of Gender Bias in Speech Translationcited by 23
  13. 2023Gender Neutralization for an Inclusive Machine Translation: from Theoretical Foundations to Open Challengescited by 22
  14. 2019Analysing Neural Language Models: Contextual Decomposition Reveals Default Reasoning in Number and Gender Assignmentcited by 20
  15. 2022Evaluating Feature Attribution Methods in the Image Domaincited by 19
  16. 2024Findings of the IWSLT 2024 Evaluation Campaigncited by 19
  17. 2023When Good and Reproducible Results are a Giant with Feet of Clay: The Importance of Software Quality in NLPcited by 14
  18. 2023Homophone Disambiguation Reveals Patterns of Context Mixing in Speech Transformerscited by 13
  19. 2023Rather a Nurse than a Physician -- Contrastive Explanations under Investigationcited by 11
  20. 2023Can We Trust Explainable AI Methods on ASR? An Evaluation on Phoneme Recognitioncited by 10
  21. 2021Let the CAT out of the bag: Contrastive Attributed explanations for Textcited by 9
  22. 2022Visualizing Automatic Speech Recognition -- Means for a Better Understanding?cited by 6
  23. 2024SPES: Spectrogram Perturbation for Explainable Speech-to-Text Generationcited by 6
  24. 2018Regional Multi-scale Approach for Visually Pleasing Explanations of Deep Neural Networkscited by 4
  25. 2023Integrating Language Models into Direct Speech Translation: An Inference-Time Solution to Control Gender Inflectioncited by 3
  26. 2017Improving coreference resolution with automatically predicted prosodic informationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Voice, Bias, and Coreference: An Interpretability Study of Gender in Speech Translationcited by 5
  2. 2025What Triggers my Model? Contrastive Explanations Inform Gender Choices by Translation Modelscited by 2
  3. 2026Explaining GAND: A Resource on Gender-Ambiguous Natural Data & Contrastive Attributioncited by 0