mlsanityData to 7 Oct 2026

Voice Memory for Agentic Speech Recognition

arXiv 2607.26410 ↗First posted 29 Jul 2026cs.CL cs.AI cs.SDcited by 0

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

We present Voice Memory, a inference-only scheme for agentic speech recognition: at stream time, a frozen corrector reads a single per-domain memory.md and decides per utterance whether to act on the hypothesis or abstain and keep the 1-best. Asynchronously, a score-gated optimizer revises that file through bounded edits, accepting an edit only when it strictly improves a held-out score. Extended from classical ASR-LM framework, we refer this split the listener-thinker architecture; the two roles are coupled only through the memory, so no weights change and the learned skill stays auditable and portable. Restraint turns out to be the operative skill this loop discovers: unconstrained generative error correction (GER) over-corrects, breaking correct tokens on up to 64% of its edits on financial news, and Voice Memory, reduces this rate to 35%. Across ten HyPoradise domains with an open corrector, Voice Memory, lowers weighted word error rate from 8.36% to 7.52% (7.47% with three added in-context examples) without regressing any dataset below its 1-best baseline; gains concentrate where recoverable headroom is largest, including air-travel commands (8.40% to 3.40%) and noisy far-field speech (CHiME-4, 12.69% to 10.46%). The memory transfers across corrector families and adds zero parameters to the inference path. A demo and example code are provided for future studies.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 43 of its 62 references that are in the map. First posted on arXiv 29 Jul 2026.

Builds on

43 of 62 references.

Closest ancestors · 1

Same direction, newest first

  1. 2026Speech-Hands: A Self-Reflection Voice Agentic Approach to Speech Recognition and Audio Reasoning with Omni Perception14 Jan

Other references · 42

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  8. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  9. 2023Large Language Models as Optimizerscited by 830
  10. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  11. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  12. 2025GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningcited by 449
30 more
  1. 2024TextGrad: Automatic "Differentiation" via Textcited by 250
  2. 2026Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skillscited by 161
  3. 2018Deep context: end-to-end contextual speech recognitioncited by 136
  4. 2026SkillOpt: Executive Strategy for Self-Evolving Agent Skillscited by 122
  5. 2023Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognitioncited by 120
  6. 2021Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusioncited by 100
  7. 2020Contextual RNN-T For Open Domain ASRcited by 85
  8. 2020Rethinking Evaluation in ASR: Are Our Models Robust Enough?cited by 72
  9. 2023Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Promptingcited by 71
  10. 2023Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognitioncited by 56
  11. 2021FastCorrect: Fast Error Correction with Edit Alignment for Automatic Speech Recognitioncited by 43
  12. 2023N-best T5: Robust ASR Error Correction using Multiple Input Hypotheses and Constrained Decoding Spacecited by 41
  13. 2023HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Modelscited by 41
  14. 2021Semantic Distance: A New Metric for ASR Performance Analysis Towards Spoken Language Understandingcited by 40
  15. 2026MiniMax Sparse Attentioncited by 34
  16. 2024Large Language Models are Efficient Learners of Noise-Robust Speech Recognitioncited by 30
  17. 2024GenTranslate: Large Language Models are Generative Multilingual Speech and Machine Translatorscited by 22
  18. 2020WER we are and WER we think we arecited by 19
  19. 2025MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokenscited by 19
  20. 2024Chain-of-Thought Prompting for Speech Translationcited by 17
  21. 2026Text-only adaptation in LLM-based ASR through text denoisingcited by 8
  22. 2026DuplexOmni: Real-Time Listening, Seeing, Thinking, and Speaking for Full-Duplex Interactioncited by 8
  23. 2024NeKo: Cross-Modality Post-Recognition Error Correction with Tasks-Guided Mixture-of-Experts Language Modelcited by 7
  24. 2022Scaling Up Deliberation for Multilingual ASRcited by 6
  25. 2024Let's Fuse Step by Step: A Generative Fusion Decoding Algorithm with LLMs for Robust and Instruction-Aware ASR and OCRcited by 4
  26. 2026Audio-Mind: An Auditable Agentic Framework for Audio Understandingcited by 2
  27. 2024LipGER: Visually-Conditioned Generative Error Correction for Robust Automatic Speech Recognitioncited by 1
  28. 2025Two Heads Are Better Than One: Audio-Visual Speech Error Correction with Dual Hypothesescited by 1
  29. 2026SpeechEQ: Benchmarking Emotional Intelligence Quotient in Socially Aware Voice Conversational Modelscited by 1
  30. 2026Preference-ASR: A Preference-Aware Test Set for Benchmarking ASR in the Era of Speech LLMscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026AgenticASR: Refining Speech Recognition in Real-World Scenarios via an Agentic Approach
  2. 2026SpeechGym: An Audio-Native Gym for Training Voice Agents via Reinforcement Learning
  3. 2026Beyond Fresh Starts: Stateful Inference for Streaming ASR in Conversational Voice Agents
  4. 2025Thinking While Speaking: Inference-Time Knowledge Transfer for Responsive and Intelligent Conversational Voice Agents
  5. 2026Towards Human-Like Interactive Speech Recognition With Agentic Correction and Semantic Evaluation
  6. 2026Anatomy of Agentic Memory: Taxonomy and Empirical Analysis of Evaluation and System Limitations
  7. 2026MemReader: From Passive to Active Extraction for Long-Term Agent Memory
  8. 2026Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontiers