mlsanityData to 14 Oct 2026

Should We Type or Talk to LLM Agents? A Comprehensive Study of Voice and Keyboard Input Perturbations

arXiv 2608.03970 ↗First posted 4 Aug 2026cs.AIcited by 0

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Human input reaches language models by typing or speaking, and each channel leaves a distinct signature: orthographic noise for keyboards; for voice, disfluency from conventional transcription and restructuring from AI-backed dictation tools. How do they impact an LLM's performance? In this paper we present HIVE (Human Input-Variation Engine), a suite of voice transcription perturbations and QWERTY keyboard perturbations. We use HIVE to evaluate how robust models are to these perturbations. We present seven findings. (i) Voice transcription perturbations lower accuracy across every instruction-tuned model we test, and it is the structure of the transcription rather than its fillers that carries the cost. (ii) QWERTY keyboard perturbations cost less, and a model absorbs a lot of them before accuracy falls away. (iii) Both trace back to one cause, how many of the question's tokens survive the perturbation: destroying a token is what hurts, while adding new ones alongside it costs little. (iv) The gap between the two channels appears only where the answer must be constructed or deduced; on multiple choice there is none. (v) The harm does not solely come from test-set contamination. (vi) It cannot be trained away with lightweight adaptation. (vii) A thinking budget recovers the keyboard channel almost entirely but leaves the spoken registers untouched, and compressed speech is worse with it.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 23 of its 31 references that are in the map. First posted on arXiv 4 Aug 2026.

Builds on

23 of 31 references.

Closest ancestors · 1

Same direction, newest first

  1. 2026Full-Duplex-Bench-v3: Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluency6 Apr

Other references · 22

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  5. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  6. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  7. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  8. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 671
  9. 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 381
  10. 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 345
  11. 2020BAE: BERT-based Adversarial Examples for Text Classificationcited by 316
  12. 2017Synthetic and Natural Noise Both Break Neural Machine Translationcited by 306
10 more
  1. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 198
  2. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 191
  3. 2019Combating Adversarial Misspellings with Robust Word Recognitioncited by 163
  4. 2022MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languagescited by 146
  5. 2021Evaluating the Robustness of Neural Language Models to Input Perturbationscited by 87
  6. 2018Spoken SQuAD: A Study of Mitigating the Impact of Speech Recognition Errors on Listening Comprehensioncited by 74
  7. 2021Disfl-QA: A Benchmark Dataset for Understanding Disfluencies in Question Answeringcited by 20
  8. 2024MEDSAGE: Enhancing Robustness of Medical Dialogue Summarization to ASR Errors with LLM-generated Synthetic Dialoguescited by 8
  9. 2025VocalBench-DF: A Benchmark for Evaluating Speech LLM Robustness to Disfluencycited by 8
  10. 2026Lost in Transcription: How Speech-to-Text Errors Derail Code Understandingcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Talk2Agent: Benchmarking Voice Interfaces for Text Agents
  2. 2024Learning to Ask: When LLM Agents Meet Unclear Instruction
  3. 2026Do Audio LLMs Listen Before They Act? Diagnosing Acoustic-Context Gating in Voice Agents
  4. 2025Spoken Conversational Agents with Large Language Models
  5. 2026Understanding Tone-Dependent Inference Cost in Large Language Models
  6. 2025Can Speech LLMs Think while Listening?
  7. 2023Evaluating Concurrent Robustness of Language Models Across Diverse Challenge Sets
  8. 2025Thinking While Speaking: Inference-Time Knowledge Transfer for Responsive and Intelligent Conversational Voice Agents