mlsanityData to 14 Oct 2026

Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction

arXiv 2502.11946 ↗First posted 17 Feb 2025cs.CL cs.AI cs.HCcited by 83

Not on the map. Why?

Abstract (arXiv)

Real-time speech interaction, serving as a fundamental interface for human-machine collaboration, holds immense potential. However, current open-source models face limitations such as high costs in voice data collection, weakness in dynamic control, and limited intelligence. To address these challenges, this paper introduces Step-Audio, the first production-ready open-source solution. Key contributions include: 1) a 130B-parameter unified speech-text multi-modal model that achieves unified understanding and generation, with the Step-Audio-Chat version open-sourced; 2) a generative speech data engine that establishes an affordable voice cloning framework and produces the open-sourced lightweight Step-Audio-TTS-3B model through distillation; 3) an instruction-driven fine control system enabling dynamic adjustments across dialects, emotions, singing, and RAP; 4) an enhanced cognitive architecture augmented with tool calling and role-playing abilities to manage complex tasks effectively. Based on our new StepEval-Audio-360 evaluation benchmark, Step-Audio achieves state-of-the-art performance in human evaluations, especially in terms of instruction following. On open-source benchmarks like LLaMA Question, shows 9.3% average performance improvement, demonstrating our commitment to advancing the development of open-source multi-modal language technologies. Our code and models are available at https://github.com/stepfun-ai/Step-Audio.

Key facts. Cited by 83 papers in the map (data through 14 Oct 2026). Builds on 19 of its 24 references that are in the map. First posted on arXiv 17 Feb 2025.

Builds on

19 of 24 references.

References · 19

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  2. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 574
  3. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  4. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  5. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 300
  6. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 288
  7. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 265
  8. 2024GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbotcited by 224
  9. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 210
  10. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 173
  11. 2022Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognitioncited by 144
  12. 2024Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLMcited by 141
7 more
  1. 2022Hybrid Transformers for Music Source Separationcited by 134
  2. 2024WavLLM: Towards Robust and Adaptive Speech Large Language Modelcited by 109
  3. 2025MinMo: A Multimodal Large Language Model for Seamless Voice Interactioncited by 79
  4. 2024SpeechVerse: A Large-scale Generalizable Audio Language Modelcited by 63
  5. 2023An Enhanced Res2Net with Local and Global Feature Fusion for Speaker Verificationcited by 42
  6. 2025LUCY: Linguistic Understanding and Control Yielding Early Stage of Hercited by 17
  7. 2024ARCON: Advancing Auto-Regressive Continuation for Driving Videoscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 51; 2026: 32.

83 papers.

Sort:

By area:

  1. 2025Kimi-Audio Technical Reportcited by 222
  2. 2025Step-Audio 2 Technical Reportcited by 105
  3. 2025Step1X-3D: Towards High-Fidelity and Controllable Generation of Textured 3D Assetscited by 87
  4. 2025Ming-Omni: A Unified Multimodal Model for Perception and Generationcited by 66
  5. 2025OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLMcited by 59
  6. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 29
  7. 2025SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversationcited by 27
  8. 2025Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generationcited by 22
  9. 2025Exploring Hallucination of Large Multimodal Models in Video Understanding: Benchmark, Analysis and Mitigationcited by 20
  10. 2026Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Musiccited by 19
  11. 2025X-Streamer: Unified Human World Modeling with Audiovisual Interactioncited by 18
  12. 2025Audio-Aware Large Language Models as Judges for Speaking Stylescited by 17
38 more
  1. 2025Empowering Time Series Analysis with Synthetic Data: A Survey and Outlook in the Era of Foundation Modelscited by 15
  2. 2025WavReward: Spoken Dialogue Models With Generalist Reward Evaluatorscited by 15
  3. 2025Chronological Thinking in Full-Duplex Spoken Dialogue Language Modelscited by 14
  4. 2025TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modelingcited by 13
  5. 2025TeleAntiFraud-28k: An Audio-Text Slow-Thinking Dataset for Telecom Fraud Detectioncited by 12
  6. 2025Towards Spoken Mathematical Reasoning: Benchmarking Speech-based Models over Multi-faceted Math Problemscited by 12
  7. 2026MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Modelscited by 12
  8. 2026MOSS-Audio Technical Reportcited by 11
  9. 2025OpenS2S: Advancing Fully Open-Source End-to-End Empathetic Large Speech Language Modelcited by 9
  10. 2025VStyle: A Benchmark for Voice Style Adaptation with Spoken Instructionscited by 9
  11. 2025DIFFA: Large Language Diffusion Models Can Listen and Understandcited by 8
  12. 2025WenetSpeech-Chuan: A Large-Scale Sichuanese Corpus with Rich Annotation for Dialectal Speech Processingcited by 8
  13. 2025From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Datacited by 7
  14. 2025Step-Audio-AQAA: a Fully End-to-End Expressive Large Audio Language Modelcited by 7
  15. 2025MGM-Omni: Scaling Omni LLMs to Personalized Long-Horizon Speechcited by 7
  16. 2025Step-Audio-EditX Technical Reportcited by 7
  17. 2025ViBES: A Conversational Agent with Behaviorally-Intelligent 3D Virtual Bodycited by 6
  18. 2026The ICASSP 2026 HumDial Challenge: Benchmarking Human-like Spoken Dialogue Systems in the LLM Eracited by 5
  19. 2025Audio Turing Test: Benchmarking the Human-likeness of Large Language Model-based Text-to-Speech Systems in Chinesecited by 4
  20. 2025ProsodyLM: Uncovering the Emerging Prosody Processing Capabilities in Speech Language Modelscited by 4
  21. 2025EchoX: Towards Mitigating Acoustic-Semantic Gap via Echo Training for Speech-to-Speech LLMscited by 4
  22. 2025StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient SpeechLLMscited by 4
  23. 2025VCB Bench: An Evaluation Benchmark for Audio-Grounded Large Language Model Conversational Agentscited by 4
  24. 2025UniMoE-Audio: Unified Speech and Music Generation with Dynamic-Capacity MoEcited by 4
  25. 2025ALMGuard: Safety Shortcuts and Where to Find Them as Guardrails for Audio-Language Modelscited by 4
  26. 2025MAViD: A Multimodal Framework for Audio-Visual Dialogue Understanding and Generationcited by 4
  27. 2025AgriGPT-Omni: A Unified Speech-Vision-Text Framework for Multilingual Agricultural Intelligencecited by 4
  28. 2026DyWeight: Dynamic Gradient Weighting for Few-Step Diffusion Samplingcited by 4
  29. 2025DIP: Efficient Large Multimodal Model Training with Dynamic Interleaved Pipelinecited by 3
  30. 2025From Text to Talk: Audio-Language Model Needs Non-Autoregressive Joint Trainingcited by 3
  31. 2026HearSay Benchmark: Do Audio LLMs Leak What They Hear?cited by 3
  32. 2026DIFFA-2: A Practical Diffusion Large Language Model for General Audio Understandingcited by 3
  33. 2026M*: A Modular, Extensible, Serving System for Multimodal Modelscited by 3
  34. 2026Multimodal Duplex Interaction Agentcited by 3
  35. 2025GOAT-SLM: A Spoken Language Model with Paralinguistic and Speaker Characteristic Awarenesscited by 2
  36. 2025VoxRole: A Comprehensive Benchmark for Evaluating Speech-Based Role-Playing Agentscited by 2
  37. 2025Extending Audio Context for Long-Form Understanding in Large Audio-Language Modelscited by 2
  38. 2025UltraVoice: Scaling Fine-Grained Style-Controlled Speech Conversations for Spoken Dialogue Modelscited by 2

50 most cited of 83 All 83 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Step-Audio-R1 Technical Report
  2. 2025Audio MultiChallenge: A Multi-Turn Evaluation of Spoken Dialogue Systems on Natural Human Interaction
  3. 2023Audiobox: Unified Audio Generation with Natural Language Prompts
  4. 2025InstructAudio: Unified speech and music generation with natural language instruction
  5. 2026Qwen-Audio-3.1-Realtime: Towards Reliable Agentic Voice Interaction
  6. 2025AU-Harness: An Open-Source Toolkit for Holistic Evaluation of Audio LLMs
  7. 2026VoiceSculptor: Your Voice, Designed By You
  8. 2026Audio2Tool: Speak, Call, Act -- A Dataset for Benchmarking Speech Tool Use