mlsanityData to 14 Oct 2026

SpeechIQ: Speech-Agentic Intelligence Quotient Across Cognitive Levels in Voice Understanding by Large Language Models

arXiv 2507.19361 ↗First posted 25 Jul 2025cs.CL cs.AI cs.SCcited by 2

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

We introduce Speech-based Intelligence Quotient (SIQ) as a new form of human cognition-inspired evaluation pipeline for voice understanding large language models, LLM Voice, designed to assess their voice understanding ability. Moving beyond popular voice understanding metrics such as word error rate (WER), SIQ examines LLM Voice across three cognitive levels motivated by Bloom's Taxonomy: (1) Remembering (i.e., WER for verbatim accuracy); (2) Understanding (i.e., similarity of LLM's interpretations); and (3) Application (i.e., QA accuracy for simulating downstream tasks). We demonstrate that SIQ not only quantifies voice understanding abilities but also provides unified comparisons between cascaded methods (e.g., ASR LLM) and end-to-end models, identifies annotation errors in existing benchmarks, and detects hallucinations in LLM Voice. Our framework represents a first-of-its-kind intelligence examination that bridges cognitive principles with voice-oriented benchmarks, while exposing overlooked challenges in multi-modal training. Our code and data will be open source to encourage future studies.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 42 of its 78 references that are in the map. First posted on arXiv 25 Jul 2025.

Builds on

42 of 78 references.

References · 42

  1. 2020Scaling Laws for Neural Language Modelscited by 6,586
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  6. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  7. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  8. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  9. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  10. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  11. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
  12. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
30 more
  1. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  2. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 217
  3. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 186
  4. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 172
  5. 2025Baichuan-Omni-1.5 Technical Reportcited by 111
  6. 2023Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speechcited by 101
  7. 2023SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translationcited by 76
  8. 2023Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation taskscited by 72
  9. 2023Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Promptingcited by 72
  10. 2018Building a Conversational Agent Overnight with Dialogue Self-Playcited by 70
  11. 2019Two-Pass End-to-End Speech Recognitioncited by 64
  12. 2023Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognitioncited by 56
  13. 2024Careless Whisper: Speech-to-Text Hallucination Harmscited by 53
  14. 2022Earnings-22: A Practical Benchmark for Accents in the Wildcited by 50
  15. 2024DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Datacited by 45
  16. 2020Class LM and word mapping for contextual biasing in End-to-End ASRcited by 42
  17. 2021Semantic Distance: A New Metric for ASR Performance Analysis Towards Spoken Language Understandingcited by 42
  18. 2024SpeechGPT-Gen: Scaling Chain-of-Information Speech Generationcited by 37
  19. 2021On the limit of English conversational speech recognitioncited by 30
  20. 2024Hallucinations in Neural Automatic Speech Recognition: Identifying Errors and Hallucinatory Modelscited by 27
  21. 2021Evaluating User Perception of Speech Recognition System Quality with Semantic Distance Metriccited by 21
  22. 2023IC3: Image Captioning by Committee Consensuscited by 20
  23. 2020End-to-End Spoken Language Understanding Without Full Transcriptscited by 19
  24. 2020WER we are and WER we think we arecited by 19
  25. 2024Just ASR + LLM? A Study on Speech Large Language Models' Ability to Identify and Understand Speaker in Spoken Dialoguecited by 16
  26. 2023Leveraging Large Language Models for Exploiting ASR Uncertaintycited by 15
  27. 2023User Interaction Patterns and Breakdowns in Conversing with LLM-Powered Voice Assistantscited by 15
  28. 2024SeMaScore : a new evaluation metric for automatic speech recognition taskscited by 12
  29. 2022H_eval: A new hybrid evaluation metric for automatic speech recognition taskscited by 9
  30. 2024Spelling Correction through Rewriting of Non-Autoregressive ASR Latticescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026The Interspeech 2026 Audio Reasoning Challenge: Evaluating Reasoning Process Quality for Audio Reasoning Models and Agentscited by 6
  2. 2026SpeechEQ: Benchmarking Emotional Intelligence Quotient in Socially Aware Voice Conversational Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Models
  2. 2025SpeechR: A Benchmark for Speech Reasoning in Large Audio-Language Models
  3. 2025Can Speech LLMs Think while Listening?
  4. 2026VoiceLongMemEval: Do Assistants Remember How You Sounded?
  5. 2026AURAL: Adaptive Latent Reasoning with Joint Chunk for Speech Language Models
  6. 2026RAIL: Rethinking Auditory Intelligence in Large Audio-Language Models with a CHC-Grounded Benchmark
  7. 2025Scaling Auditory Cognition via Test-Time Compute in Audio Language Models
  8. 2026Speech-Hands: A Self-Reflection Voice Agentic Approach to Speech Recognition and Audio Reasoning with Omni Perception