# No Detectable Change in Side-Level WER from Prompt-Level Context: A Preregistered Ablation on a Production Oral-History Corpus arXiv: https://arxiv.org/abs/2608.28875 ยท HTML: https://mlsanity.com/paper/2608.28875/ ## Key facts - No citing papers in the map yet (data through 14 Oct 2026). - Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. - Builds on 28 of its 50 references that are in the map. - First posted on arXiv 28 Aug 2026. ## Placed in - Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/) ## Builds on (most cited first, 28 of 28) - [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022) - [Deep context: end-to-end contextual speech recognition](https://mlsanity.com/paper/1808.02480/) (2018) - [CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordings](https://mlsanity.com/paper/2004.09249/) (2020) - [Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusion](https://mlsanity.com/paper/2104.02194/) (2021) - [Earnings-21: A Practical Benchmark for ASR in the Wild](https://mlsanity.com/paper/2104.11348/) (2021) - [Contextual Adapters for Personalized Speech Recognition in Neural Transducers](https://mlsanity.com/paper/2205.13660/) (2022) - [Careless Whisper: Speech-to-Text Hallucination Harms](https://mlsanity.com/paper/2402.08021/) (2024) - [Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalization](https://mlsanity.com/paper/2305.11095/) (2023) - [Semantic Distance: A New Metric for ASR Performance Analysis Towards Spoken Language Understanding](https://mlsanity.com/paper/2104.02138/) (2021) - [Can Whisper perform speech-based in-context learning?](https://mlsanity.com/paper/2309.07081/) (2023) - [Tree-constrained Pointer Generator for End-to-end Contextual Speech Recognition](https://mlsanity.com/paper/2109.00627/) (2021) - [Contextual Speech Recognition with Difficult Negative Training Examples](https://mlsanity.com/paper/1810.12170/) (2018) - [Preregistering NLP Research](https://mlsanity.com/paper/2103.06944/) (2021) - [Evaluating User Perception of Speech Recognition System Quality with Semantic Distance Metric](https://mlsanity.com/paper/2110.05376/) (2021) - [Can Contextual Biasing Remain Effective with Whisper and GPT-2?](https://mlsanity.com/paper/2306.01942/) (2023) - [WER we are and WER we think we are](https://mlsanity.com/paper/2010.03432/) (2020) - [PromptASR for contextualized ASR with controllable style](https://mlsanity.com/paper/2309.07414/) (2023) - [ContextASR-Bench: A Massive Contextual Speech Recognition Benchmark](https://mlsanity.com/paper/2507.05727/) (2025) - [Zero-shot Domain-sensitive Speech Recognition with Prompt-conditioning Fine-tuning](https://mlsanity.com/paper/2307.10274/) (2023) - [Do Prompts Really Prompt? Exploring the Prompt Understanding Capability of Whisper](https://mlsanity.com/paper/2406.05806/) (2024) - [BR-ASR: Efficient and Scalable Bias Retrieval Framework for Contextual Biasing ASR in Speech LLM](https://mlsanity.com/paper/2505.19179/) (2025) - [A Two-Sided Discussion of Preregistration of NLP Research](https://mlsanity.com/paper/2302.10086/) (2023) - [Keyword-Guided Adaptation of Automatic Speech Recognition](https://mlsanity.com/paper/2406.02649/) (2024) - [Improving Child Speech Recognition and Reading Mistake Detection by Using Prompts](https://mlsanity.com/paper/2506.11079/) (2025) - [PROFASR-BENCH: A Benchmark for Context-Conditioned ASR in High-Stakes Professional Speech](https://mlsanity.com/paper/2512.23686/) (2025) - [IndicContextEval: A Benchmark for Evaluating Context Utilisation in Audio Large Language Models Across 8 Indic Languages](https://mlsanity.com/paper/2606.19157/) (2026) - [Contextual Earnings-22: A Speech Recognition Benchmark with Custom Vocabulary in the Wild](https://mlsanity.com/paper/2604.07354/) (2026) - [Single-Round Vector RAG vs an LLM-Compiled Wiki: A Preregistered Comparison on a Small Multi-Domain Research Corpus](https://mlsanity.com/paper/2605.18490/) (2026) ## Abstract (arXiv) Supplying context at inference time to a large multimodal model is an inexpensive lever for adapting speech transcription to a domain, and earlier results on smaller models reported large gains. This work tested that mechanism where it ships, in the prompt-conditioning layer of a production oral-history transcription tool, on a sample from its own production corpus. Full prompt-level context did not detectably change side-level word error rate (WER), and none of the four preregistered hypotheses was supported. The design was a within-item paired ablation, preregistered with the analysis code frozen by hash before the confirmatory batch was scored; two disclosed gpt-4o pilot sides had been scored earlier, during scorer development. Nineteen cassette sides, about 10.6 hours of degraded 1970s-80s interview audio, were reprocessed through the production code path under three prompt arms, crossed with two deployed commercial configurations, gpt-4o-transcribe and gemini-2.5-flash, and scored against operator-corrected verbatim references. For gpt-4o-transcribe the median paired difference between the full-context and no-context arms was +0.6 WER points, with a side-resampled interval of [-1.1, +1.0]; the Gemini estimates were too unstable to support a comparable negative inference. A post-hoc rerun found run-to-run pipeline variability larger than the confirmatory differences, so effects of that size cannot be resolved from one transcription per cell. An implementation audit verified the manipulation was live, and sequence-alignment analysis found a small improvement on complete context-listed phrases, too small to materially change side-level WER, and for Gemini coexisting with worsened unlisted-token error. Evaluating context mechanisms therefore requires sequence-aligned term-level, insertion, and speaker-label measures alongside aggregate accuracy.