mlsanityData to 7 Oct 2026

GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbot

arXiv 2412.02612 ↗First posted 3 Dec 2024cs.CL cs.SD eess.AScited by 218

Not on the map. Why?

Cited by 25 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce GLM-4-Voice, an intelligent and human-like end-to-end spoken chatbot. It supports both Chinese and English, engages in real-time voice conversations, and varies vocal nuances such as emotion, intonation, speech rate, and dialect according to user instructions. GLM-4-Voice uses an ultra-low bitrate (175bps), single-codebook speech tokenizer with 12.5Hz frame rate derived from an automatic speech recognition (ASR) model by incorporating a vector-quantized bottleneck into the encoder. To efficiently transfer knowledge from text to speech modalities, we synthesize speech-text interleaved data from existing text pre-training corpora using a text-to-token model. We continue pre-training from the pre-trained text language model GLM-4-9B with a combination of unsupervised speech data, interleaved speech-text data, and supervised speech-text data, scaling up to 1 trillion tokens, achieving state-of-the-art performance in both speech language modeling and spoken question answering. We then fine-tune the pre-trained model with high-quality conversational speech data, achieving superior performance compared to existing baselines in both conversational ability and speech quality. The open models can be accessed through https://github.com/THUDM/GLM-4-Voice and https://huggingface.co/THUDM/glm-4-voice-9b.

Key facts. Cited by 218 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Dec 2024.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 115; 2026: 102.

218 papers.

Sort:

By area:

  1. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
  2. 2025Kimi-Audio Technical Reportcited by 210
  3. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  4. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  5. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  6. 2025MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmarkcited by 101
  7. 2025Step-Audio 2 Technical Reportcited by 101
  8. 2025Step-Audio: Unified Understanding and Generation in Intelligent Speech Interactioncited by 83
  9. 2025MinMo: A Multimodal Large Language Model for Seamless Voice Interactioncited by 79
  10. 2025LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesiscited by 75
  11. 2025Baichuan-Audio: A Unified Framework for End-to-End Speech Interactioncited by 68
  12. 2025MiMo-Audio: Audio Language Models are Few-Shot Learnerscited by 58
38 more
  1. 2026PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Modelscited by 57
  2. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  3. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 47
  4. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  5. 2025SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Modelcited by 32
  6. 2025VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Modelcited by 30
  7. 2025URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Modelscited by 29
  8. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 27
  9. 2025SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversationcited by 25
  10. 2025STITCH: Simultaneous Thinking and Talking with Chunked Reasoning for Spoken Language Modelscited by 25
  11. 2025XY-Tokenizer: Mitigating the Semantic-Acoustic Conflict in Low-Bitrate Speech Codecscited by 21
  12. 2025Mini-Omni-Reasoner: Token-Level Thinking-in-Speaking in Large Speech Modelscited by 20
  13. 2025MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Modelscited by 20
  14. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matchingcited by 19
  15. 2025Audio-Aware Large Language Models as Judges for Speaking Stylescited by 17
  16. 2025Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Modelcited by 17
  17. 2025FD-Bench: A Full-Duplex Benchmarking Pipeline Designed for Full Duplex Spoken Dialogue Systemscited by 17
  18. 2025Fun-Audio-Chat Technical Reportcited by 17
  19. 2025VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Modelscited by 16
  20. 2025S2S-Arena: Evaluating Paralinguistic Instruction Following in Speech-to-Speech Modelscited by 16
  21. 2025VocalNet: Speech LLM with Multi-Token Prediction for Faster and High-Quality Generationcited by 16
  22. 2025AudioJailbreak: Jailbreak Attacks against End-to-End Large Audio-Language Modelscited by 16
  23. 2025X-Streamer: Unified Human World Modeling with Audiovisual Interactioncited by 16
  24. 2025FlexDuo: A Pluggable System for Enabling Full-Duplex Capabilities in Speech Dialogue Systemscited by 15
  25. 2025TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modelingcited by 15
  26. 2025WavReward: Spoken Dialogue Models With Generalist Reward Evaluatorscited by 15
  27. 2025VocalBench: Benchmarking the Vocal Conversational Abilities for Speech Interaction Modelscited by 15
  28. 2025Ming-UniAudio: Speech LLM for Joint Understanding, Generation and Editing with Unified Representationcited by 15
  29. 2025OmniCharacter: Towards Immersive Role-Playing Agents with Seamless Speech-Language Personality Interactioncited by 14
  30. 2025MTalk-Bench: Evaluating Speech-to-Speech Models in Multi-Turn Dialogues via Arena-style and Rubrics Protocolscited by 14
  31. 2025Chronological Thinking in Full-Duplex Spoken Dialogue Language Modelscited by 13
  32. 2025TeleAntiFraud-28k: An Audio-Text Slow-Thinking Dataset for Telecom Fraud Detectioncited by 12
  33. 2025TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modelingcited by 12
  34. 2025InSerter: Speech Instruction Following with Unsupervised Interleaved Pre-trainingcited by 11
  35. 2025Analyzing Mitigation Strategies for Catastrophic Forgetting in End-to-End Training of Spoken Language Modelscited by 11
  36. 2025JALMBench: Benchmarking Jailbreak Vulnerabilities in Audio Language Modelscited by 11
  37. 2025WildSpeech-Bench: Benchmarking End-to-End SpeechLLMs in the Wildcited by 11
  38. 2025Easy Turn: Integrating Acoustic and Linguistic Modalities for Robust Turn-Taking in Full-Duplex Spoken Dialogue Systemscited by 11

50 most cited of 218 All 218 (JSON)