mlsanityData to 7 Oct 2026

Audio Interaction Model

arXiv 2606.05121 ↗First posted 3 Jun 2026cs.SD cs.AI cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Audio is continuous and interactive, yet most Large Audio Language Models (LALMs) remain offline and streaming systems usually specialize in ASR or spoken dialogue. We formalize the Audio Interaction Model, an always-on perceive--decide--respond paradigm that tracks context, decides whether intervention is warranted, and responds without stopping listening. We instantiate it with Audio-Interaction and introduce SoundFlow, coupling streaming-native data construction, comprehension-aware silence/response supervision, dual-loss training, and asynchronous FIFO inference. We also construct textsc{StreamAudio-2M, a 2.6M-item, 302k-hour corpus spanning 7 capability families and 28 sub-tasks, together with Proactive-Sound-Bench. Across 8 benchmarks, Audio-Interaction remains competitive on mainstream audio tasks while enabling spoken-instruction robustness, long-stream interaction, and proactive intervention.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 56 of its 62 references that are in the map. First posted on arXiv 3 Jun 2026.

Builds on

56 of 62 references.

References · 56

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  9. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  10. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  11. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  12. 2025Qwen2.5-Omni Technical Reportcited by 704
38 more
  1. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  2. 2024Qwen2-Audio Technical Reportcited by 543
  3. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  4. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  5. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
  6. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 400
  7. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 281
  8. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 216
  9. 2025Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Modelscited by 210
  10. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 204
  11. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 192
  12. 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 190
  13. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 183
  14. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 169
  15. 2019WHAM!: Extending Speech Separation to Noisy Environmentscited by 154
  16. 2022Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognitioncited by 139
  17. 2024Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLMcited by 135
  18. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 127
  19. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  20. 2026Qwen3-ASR Technical Reportcited by 105
  21. 2024Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilitiescited by 104
  22. 2025MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmarkcited by 101
  23. 2025Step-Audio 2 Technical Reportcited by 101
  24. 2025Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Modelscited by 94
  25. 2023Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLMcited by 92
  26. 2025LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesiscited by 75
  27. 2024Language Model Can Listen While Speakingcited by 62
  28. 2024Seed-ASR: Understanding Diverse Speech and Contexts with LLM-based Speech Recognitioncited by 61
  29. 2025Canary-1B-v2 & Parakeet-TDT-0.6B-v3: Efficient and High-Performance Models for Multilingual ASR and ASTcited by 51
  30. 2025FireRedASR: Open-Source Industrial-Grade Mandarin Speech Recognition Models from Encoder-Decoder to LLM Integrationcited by 44
  31. 2021SD-QA: Spoken Dialectal Question Answering for the Real Worldcited by 36
  32. 2026Proactive Agent Research Environment: Simulating Active Users to Evaluate Proactive Assistantscited by 14
  33. 2025Chronological Thinking in Full-Duplex Spoken Dialogue Language Modelscited by 13
  34. 2026EmotionThinker: Prosody-Aware Reinforcement Learning for Explainable Speech Emotion Reasoningcited by 13
  35. 2023The Intel Neuromorphic DNS Challengecited by 9
  36. 2025Thinking with Sound: Audio Chain-of-Thought Enables Multimodal Reasoning in Large Audio-Language Modelscited by 9
  37. 2026PASK: Toward Intent-Aware Proactive Agents with Long-Term Memorycited by 8
  38. 2026DuplexSLA: A Full-Duplex Spoken Language Model with Synchronized Speech, Language, and Actioncited by 8

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Multimodal Duplex Interaction Agentcited by 3
  2. 2026VoiceMem: Streaming Dual-Brain Memory for Real-Time Interactioncited by 1
  3. 2026TimeInteract: Towards Real-Time Interactive Intelligence for Streaming Time Seriescited by 0