mlsanityData to 14 Oct 2026

Building a Taiwanese Mandarin Spoken Language Model: A First Attempt

arXiv 2411.07111 ↗First posted 11 Nov 2024cs.CL cs.SD eess.AScited by 23

Not on the map. Why?

Abstract (arXiv)

This technical report presents our initial attempt to build a spoken large language model (LLM) for Taiwanese Mandarin, specifically tailored to enable real-time, speech-to-speech interaction in multi-turn conversations. Our end-to-end model incorporates a decoder-only transformer architecture and aims to achieve seamless interaction while preserving the conversational flow, including full-duplex capabilities allowing simultaneous speaking and listening. The paper also details the training process, including data preparation with synthesized dialogues and adjustments for real-time interaction. We also developed a platform to evaluate conversational fluency and response coherence in multi-turn dialogues. We hope the release of the report can contribute to the future development of spoken LLMs in Taiwanese Mandarin.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 28 of its 31 references that are in the map. First posted on arXiv 11 Nov 2024.

Builds on

28 of 31 references.

References · 28

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2024GPT-4o System Cardcited by 4,588
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  6. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  7. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  8. 2018VoxCeleb2: Deep Speaker Recognitioncited by 875
  9. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  10. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  11. 2017VoxCeleb: a large-scale speaker identification datasetcited by 732
  12. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 628
16 more
  1. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  2. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  3. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  4. 2019NeMo: a toolkit for building AI applications using Neural Modulescited by 235
  5. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 210
  6. 2020Attention is All You Need in Speech Separationcited by 185
  7. 2022Generative Spoken Dialogue Language Modelingcited by 126
  8. 2023Powerset multi-class cross entropy loss for neural speaker diarizationcited by 111
  9. 2023Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speechcited by 101
  10. 2024Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Taskscited by 71
  11. 2024Language Model Can Listen While Speakingcited by 63
  12. 2024Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversationscited by 45
  13. 2021Deep Learning Based Assessment of Synthetic Speech Naturalnesscited by 35
  14. 2023Turning Whisper into Real-Time Transcription Systemcited by 24
  15. 2024Human Latency Conversational Turns for Spoken Avatar Systemscited by 5
  16. 2024Leave No Knowledge Behind During Knowledge Distillation: Towards Practical and Effective Knowledge Distillation for Code-Switching ASR Using Realistic Datacited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 11.

23 papers.

Sort:

By area:

  1. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 50
  2. 2025SAKURA: On the Multi-hop Reasoning of Large Audio-Language Models Based on Speech and Audio Informationcited by 42
  3. 2025Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Surveycited by 39
  4. 2025FD-Bench: A Full-Duplex Benchmarking Pipeline Designed for Full Duplex Spoken Dialogue Systemscited by 20
  5. 2025Analyzing Mitigation Strategies for Catastrophic Forgetting in End-to-End Training of Spoken Language Modelscited by 11
  6. 2026All That Glitters Is Not Audio: Rethinking Text Priors and Audio Reliance in Audio-Language Evaluationcited by 8
  7. 2026AQUA-Bench: Beyond Finding Answers to Knowing When There Are None in Audio Question Answeringcited by 6
  8. 2026How Contrastive Decoding Enhances Large Audio Language Modelscited by 6
  9. 2025ToxicTone: A Mandarin Audio Dataset Annotated for Toxicity and Toxic Utterance Tonalitycited by 5
  10. 2025SAKE: Towards Editing Auditory Attribute Knowledge of Large Audio-Language Modelscited by 5
  11. 2025Investigating Safety Vulnerabilities of Large Audio-Language Models Under Speaker Emotional Variationscited by 4
  12. 2026Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Modelscited by 4
11 more
  1. 2025CantoASR: Prosody-Aware ASR-LALM Collaboration for Low-Resource Cantonesecited by 3
  2. 2026ASPIRin: Action Space Projection for Interactivity-Optimized Reinforcement Learning in Full-Duplex Speech Language Modelscited by 3
  3. 2025An Exploration of Mamba for Speech Self-Supervised Modelscited by 2
  4. 2026MUGEN: Evaluating and Improving Multi-audio Understanding of Large Audio-Language Modelscited by 2
  5. 2026Nudging Hidden States: Training-Free Model Steering for Chain-of-Thought Reasoning in Large Audio-Language Modelscited by 2
  6. 2026How Auditory Knowledge in LLM Backbones Shapes Audio Language Models: A Holistic Evaluationcited by 2
  7. 2026CAAD: Contrastive Audio-Aware Distillation for Efficient Speech Language Modelscited by 2
  8. 2025TAU: A Benchmark for Cultural Sound Understanding Beyond Semanticscited by 1
  9. 2025Pseudo2Real: Task Arithmetic for Pseudo-Label Correction in Automatic Speech Recognitioncited by 1
  10. 2026The Sound of Absence: Audio-Language Embedding Models Struggle with Negationcited by 1
  11. 2026Encoder-Side Neuron Identification and Amplification for Acoustic Perception in Large Audio-Language Modelscited by 0