mlsanityData to 7 Oct 2026

No Detectable Change in Side-Level WER from Prompt-Level Context: A Preregistered Ablation on a Production Oral-History Corpus

arXiv 2608.28875 ↗First posted 28 Aug 2026cs.CL cs.AI cs.SDcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Supplying context at inference time to a large multimodal model is an inexpensive lever for adapting speech transcription to a domain, and earlier results on smaller models reported large gains. This work tested that mechanism where it ships, in the prompt-conditioning layer of a production oral-history transcription tool, on a sample from its own production corpus. Full prompt-level context did not detectably change side-level word error rate (WER), and none of the four preregistered hypotheses was supported. The design was a within-item paired ablation, preregistered with the analysis code frozen by hash before the confirmatory batch was scored; two disclosed gpt-4o pilot sides had been scored earlier, during scorer development. Nineteen cassette sides, about 10.6 hours of degraded 1970s-80s interview audio, were reprocessed through the production code path under three prompt arms, crossed with two deployed commercial configurations, gpt-4o-transcribe and gemini-2.5-flash, and scored against operator-corrected verbatim references. For gpt-4o-transcribe the median paired difference between the full-context and no-context arms was +0.6 WER points, with a side-resampled interval of [-1.1, +1.0]; the Gemini estimates were too unstable to support a comparable negative inference. A post-hoc rerun found run-to-run pipeline variability larger than the confirmatory differences, so effects of that size cannot be resolved from one transcription per cell. An implementation audit verified the manipulation was live, and sequence-alignment analysis found a small improvement on complete context-listed phrases, too small to materially change side-level WER, and for Gemini coexisting with worsened unlisted-token error. Evaluating context mechanisms therefore requires sequence-aligned term-level, insertion, and speaker-label measures alongside aggregate accuracy.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 50 references that are in the map. First posted on arXiv 28 Aug 2026.

Builds on

28 of 50 references.

References · 28

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  2. 2018Deep context: end-to-end contextual speech recognitioncited by 136
  3. 2020CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordingscited by 129
  4. 2021Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusioncited by 100
  5. 2021Earnings-21: A Practical Benchmark for ASR in the Wildcited by 60
  6. 2022Contextual Adapters for Personalized Speech Recognition in Neural Transducerscited by 57
  7. 2024Careless Whisper: Speech-to-Text Hallucination Harmscited by 53
  8. 2023Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalizationcited by 43
  9. 2021Semantic Distance: A New Metric for ASR Performance Analysis Towards Spoken Language Understandingcited by 40
  10. 2023Can Whisper perform speech-based in-context learning?cited by 35
  11. 2021Tree-constrained Pointer Generator for End-to-end Contextual Speech Recognitioncited by 32
  12. 2018Contextual Speech Recognition with Difficult Negative Training Examplescited by 28
16 more
  1. 2021Preregistering NLP Researchcited by 24
  2. 2021Evaluating User Perception of Speech Recognition System Quality with Semantic Distance Metriccited by 21
  3. 2023Can Contextual Biasing Remain Effective with Whisper and GPT-2?cited by 21
  4. 2020WER we are and WER we think we arecited by 19
  5. 2023PromptASR for contextualized ASR with controllable stylecited by 18
  6. 2025ContextASR-Bench: A Massive Contextual Speech Recognition Benchmarkcited by 14
  7. 2023Zero-shot Domain-sensitive Speech Recognition with Prompt-conditioning Fine-tuningcited by 13
  8. 2024Do Prompts Really Prompt? Exploring the Prompt Understanding Capability of Whispercited by 12
  9. 2025BR-ASR: Efficient and Scalable Bias Retrieval Framework for Contextual Biasing ASR in Speech LLMcited by 10
  10. 2023A Two-Sided Discussion of Preregistration of NLP Researchcited by 6
  11. 2024Keyword-Guided Adaptation of Automatic Speech Recognitioncited by 5
  12. 2025Improving Child Speech Recognition and Reading Mistake Detection by Using Promptscited by 3
  13. 2025PROFASR-BENCH: A Benchmark for Context-Conditioned ASR in High-Stakes Professional Speechcited by 3
  14. 2026IndicContextEval: A Benchmark for Evaluating Context Utilisation in Audio Large Language Models Across 8 Indic Languagescited by 3
  15. 2026Contextual Earnings-22: A Speech Recognition Benchmark with Custom Vocabulary in the Wildcited by 2
  16. 2026Single-Round Vector RAG vs an LLM-Compiled Wiki: A Preregistered Comparison on a Small Multi-Domain Research Corpuscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Rethinking Human-Aligned Evaluation: An Analysis of Semantic Metrics Beyond WER
  2. 2026Context-Grounding Gains Are Mediated by Pre-existing Machinery: Auditing GRPO, SFT, and DPO
  3. 2026From Oracle to Noisy Context: Mitigating Contextual Exposure Bias in Speech-LLMs
  4. 2025REHEARSE: Experiential Rehearsal for Verbal Confidence Calibration in Large Language Models
  5. 2026Beyond Token Revision: Investigating Mask-and-Replace Diffusion for Zero-Shot Text-to-Speech
  6. 2025Swiss Parliaments Corpus Re-Imagined (SPC_R): Enhanced Transcription with RAG-based Correction and Predicted BLEU
  7. 2024Self-Train Before You Transcribe
  8. 2024The Surprising Effectiveness of Test-Time Training for Few-Shot Learning