mlsanityData to 7 Oct 2026

Speech-Copilot: Leveraging Large Language Models for Speech Processing via Task Decomposition, Modularization, and Program Generation

arXiv 2407.09886 ↗First posted 13 Jul 2024eess.AS cs.CL cs.SDcited by 27

Not on the map. Why?

Abstract (arXiv)

In this work, we introduce Speech-Copilot, a modular framework for instruction-oriented speech-processing tasks that minimizes human effort in toolset construction. Unlike end-to-end methods using large audio-language models, Speech-Copilot builds speech processing-specific toolsets by analyzing pre-collected task instructions and breaking tasks into manageable sub-tasks. It features a flexible agent based on large language models that performs tasks through program generation. Our approach achieves state-of-the-art performance on the Dynamic-SUPERB benchmark, demonstrating its effectiveness across diverse speech-processing tasks. Key contributions include: 1) developing an innovative framework for speech processing-specific toolset construction, 2) establishing a high-performing agent based on large language models, and 3) offering a new perspective on addressing challenging instruction-oriented speech-processing tasks. Without additional training processes required by end-to-end approaches, our method provides a flexible and extendable solution for a wide range of speech-processing applications.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 52 of its 57 references that are in the map. First posted on arXiv 13 Jul 2024.

Builds on

52 of 57 references.

References · 52

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  7. 2022Emergent Abilities of Large Language Modelscited by 2,483
  8. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  9. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  10. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  11. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  12. 2022Large Language Models Can Self-Improvecited by 656
38 more
  1. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  2. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  3. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
  4. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  5. 2024Understanding the planning of LLM agents: A surveycited by 414
  6. 2023On the Planning Abilities of Large Language Models : A Critical Investigationcited by 308
  7. 2023Large Language Models as Tool Makerscited by 293
  8. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  9. 2019NeMo: a toolkit for building AI applications using Neural Modulescited by 230
  10. 2023Listen, Think, and Understandcited by 206
  11. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 196
  12. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 183
  13. 2023SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoningcited by 157
  14. 2024EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instructioncited by 153
  15. 2024StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Modelscited by 129
  16. 2023CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsetscited by 122
  17. 2024AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Callscited by 121
  18. 2023Powerset multi-class cross entropy loss for neural speaker diarizationcited by 109
  19. 2024WavLLM: Towards Robust and Adaptive Speech Large Language Modelcited by 107
  20. 2023Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speechcited by 99
  21. 2023Joint Audio and Speech Understandingcited by 99
  22. 2023CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Modelscited by 90
  23. 2024Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarioscited by 85
  24. 2024Can Large Language Model Agents Simulate Human Trust Behavior?cited by 62
  25. 2024Learning to Use Tools via Cooperative and Interactive Agentscited by 54
  26. 2024Risks of AI Scientists: Prioritizing Safeguarding Over Autonomycited by 49
  27. 2021The Zero Resource Speech Challenge 2021: Spoken language modellingcited by 47
  28. 2024RoTBench: A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learningcited by 45
  29. 2023Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalizationcited by 43
  30. 2024DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignmentcited by 32
  31. 2024Understanding Sounds, Missing the Questions: The Challenge of Object Hallucination in Large Audio-Language Modelscited by 30
  32. 2024AI-Augmented Predictions: LLM Assistants Improve Human Forecasting Accuracycited by 21
  33. 2022Brouhaha: multi-task training for voice activity detection, speech-to-noise ratio, and C50 room acoustics estimationcited by 20
  34. 2023CommonAccent: Exploring Large Acoustic Pretrained Models for Accent Classification Based on Common Voicecited by 19
  35. 2023A Closer Look into Automatic Evaluation Using Large Language Modelscited by 17
  36. 2023LyricWhiz: Robust Multilingual Zero-shot Lyrics Transcription by Whispering to ChatGPTcited by 15
  37. 2023Investigating Zero-Shot Generalizability on Mandarin-English Code-Switched ASR and Speech-to-text Translation of Recent Foundation Models with Self-Supervision and Weak Supervisioncited by 13
  38. 2023Zero Resource Code-switched Speech Benchmark Using Speech Utterance Pairs For Multiple Spoken Languagescited by 12

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 11; 2026: 12.

27 papers.

Sort:

By area:

  1. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 47
  2. 2024DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Datacited by 43
  3. 2025SAKURA: On the Multi-hop Reasoning of Large Audio-Language Models Based on Speech and Audio Informationcited by 42
  4. 2025Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Surveycited by 38
  5. 2024Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoningcited by 29
  6. 2026Full-Duplex-Bench-v3: Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluencycited by 29
  7. 2025Speech-IFEval: Evaluating Instruction-Following and Quantifying Catastrophic Forgetting in Speech-Aware Language Modelscited by 27
  8. 2025Creativity in LLM-based Multi-Agent Systems: A Surveycited by 23
  9. 2025AudioLens: A Closer Look at Auditory Attribute Perception of Large Audio-Language Modelscited by 18
  10. 2025Teaching Audio-Aware Large Language Models What Does Not Hear: Mitigating Hallucinations through Synthesized Negative Samplescited by 11
  11. 2026AQAScore: Evaluating Semantic Alignment in Text-to-Audio Generation via Audio Question Answeringcited by 8
  12. 2025From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Datacited by 7
15 more
  1. 2026AQUA-Bench: Beyond Finding Answers to Knowing When There Are None in Audio Question Answeringcited by 6
  2. 2025SAKE: Towards Editing Auditory Attribute Knowledge of Large Audio-Language Modelscited by 5
  3. 2026All That Glitters Is Not Audio: Rethinking Text Priors and Audio Reliance in Audio-Language Evaluationcited by 5
  4. 2025Investigating Safety Vulnerabilities of Large Audio-Language Models Under Speaker Emotional Variationscited by 4
  5. 2026Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Modelscited by 4
  6. 2024Library Learning Doesn't: The Curious Case of the Single-Use "Library"cited by 3
  7. 2024Generating a Low-code Complete Workflow via Task Decomposition and RAGcited by 3
  8. 2025Audio-Maestro: Enhancing Large Audio-Language Models with Tool-Augmented Reasoningcited by 3
  9. 2026ASPIRin: Action Space Projection for Interactivity-Optimized Reinforcement Learning in Full-Duplex Speech Language Modelscited by 3
  10. 2026MUGEN: Evaluating and Improving Multi-audio Understanding of Large Audio-Language Modelscited by 2
  11. 2026How Auditory Knowledge in LLM Backbones Shapes Audio Language Models: A Holistic Evaluationcited by 2
  12. 2026Nudging Hidden States: Training-Free Model Steering for Chain-of-Thought Reasoning in Large Audio-Language Modelscited by 1
  13. 2026The Sound of Absence: Audio-Language Embedding Models Struggle with Negationcited by 1
  14. 2026Encoder-Side Neuron Identification and Amplification for Acoustic Perception in Large Audio-Language Modelscited by 0
  15. 2026Improving Text-to-Audio Instruction Following via Fine-Grained Feedback from Audio-Aware Large Language Modelscited by 0