mlsanityData to 14 Oct 2026

Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis

arXiv 2502.04128 ↗First posted 6 Feb 2025eess.AS cs.AI cs.CLcited by 70

On the map: LLM reasoning → Audio-language model reasoning

Abstract (arXiv)

Recent advances in text-based large language models (LLMs), particularly in the GPT series and the o1 model, have demonstrated the effectiveness of scaling both training-time and inference-time compute. However, current state-of-the-art TTS systems leveraging LLMs are often multi-stage, requiring separate models (e.g., diffusion models after LLM), complicating the decision of whether to scale a particular model during training or testing. This work makes the following contributions: First, we explore the scaling of train-time and inference-time compute for speech synthesis. Second, we propose a simple framework Llasa for speech synthesis that employs a single-layer vector quantizer (VQ) codec and a single Transformer architecture to fully align with standard LLMs such as Llama. Our experiments reveal that scaling train-time compute for Llasa consistently improves the naturalness of synthesized speech and enables the generation of more complex and accurate prosody patterns. Furthermore, from the perspective of scaling inference-time compute, we employ speech understanding models as verifiers during the search, finding that scaling inference-time compute shifts the sampling modes toward the preferences of specific verifiers, thereby improving emotional expressiveness, timbre consistency, and content accuracy. In addition, we released the checkpoint and training code for our TTS model (1B, 3B, 8B) and codec model publicly available.

Key facts. Cited by 70 papers in the map (data through 14 Oct 2026). Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning. Builds on 38 of its 65 references that are in the map. First posted on arXiv 6 Feb 2025.

Builds on

38 of 65 references.

References · 38

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  6. 2024OpenAI o1 System Cardcited by 1,244
  7. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  8. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  9. 2022High Fidelity Neural Audio Compressioncited by 687
  10. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  11. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  12. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
26 more
  1. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 390
  2. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 361
  3. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  4. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 300
  5. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 288
  6. 2025Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Stepscited by 234
  7. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 205
  8. 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 192
  9. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 161
  10. 2022Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognitioncited by 144
  11. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 129
  12. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 122
  13. 2023FunASR: A Fundamental End-to-End Speech Recognition Toolkitcited by 95
  14. 2024BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of datacited by 75
  15. 2024Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Modelcited by 72
  16. 2024FireRedTTS: A Foundation Text-To-Speech Framework for Industry-Level Generative Speech Applicationscited by 68
  17. 2024Autoregressive Speech Synthesis without Vector Quantizationcited by 65
  18. 2024ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reorderingcited by 50
  19. 2024Scaling Transformers for Low-Bitrate High-Quality Speech Codingcited by 49
  20. 2024BigCodec: Pushing the Limits of Low-Bitrate Neural Speech Codeccited by 46
  21. 2024SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Soundcited by 45
  22. 2024WenetSpeech4TTS: A 12,800-hour Mandarin TTS Corpus for Large Speech Generation Model Benchmarkcited by 40
  23. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  24. 2024VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignmentcited by 38
  25. 2024KALL-E:Autoregressive Speech Synthesis with Next-Distribution Predictioncited by 5
  26. 2023Diverse and Expressive Speech Prosody Prediction with Denoising Diffusion Probabilistic Modelcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 36; 2026: 34.

70 papers.

Sort:

By area:

  1. 2025Kimi-Audio Technical Reportcited by 222
  2. 2026Qwen3-TTS Technical Reportcited by 118
  3. 2025MiMo-Audio: Audio Language Models are Few-Shot Learnerscited by 65
  4. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  5. 2025VibeVoice Technical Reportcited by 29
  6. 2026OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Modelscited by 25
  7. 2025A Survey of Slow Thinking-based Reasoning LLMs using Reinforced Learning and Inference-time Scaling Lawcited by 19
  8. 2026MOSS-TTS Technical Reportcited by 14
  9. 2025TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modelingcited by 13
  10. 2025CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speechcited by 11
  11. 2025Towards Interactive Intelligence for Digital Humanscited by 10
  12. 2025FocalCodec-Stream: Streaming Low-Bitrate Speech Coding via Causal Distillationcited by 9
38 more
  1. 2025DMOSpeech 2: Reinforcement Learning for Duration Prediction in Metric-Optimized Speech Synthesiscited by 8
  2. 2025WenetSpeech-Chuan: A Large-Scale Sichuanese Corpus with Rich Annotation for Dialectal Speech Processingcited by 8
  3. 2025MagiCodec: Simple Masked Gaussian-Injected Codec for High-Fidelity Reconstruction and Generationcited by 7
  4. 2025J1: Exploring Simple Test-Time Scaling for LLM-as-a-Judgecited by 5
  5. 2025VoXtream: Full-Stream Text-to-Speech with Extremely Low Latencycited by 5
  6. 2026Dynin-Omni: Omnimodal Unified Large Diffusion Language Modelcited by 5
  7. 2026FlowTTS-GRPO: Online Reinforcement Learning with Multi-Objective Reward Optimization for Flow-Matching Based Text-to-Speechcited by 5
  8. 2025Prompt-Unseen-Emotion: Zero-shot Expressive Speech Synthesis with Prompt-LLM Contextual Knowledge for Mixed Emotionscited by 4
  9. 2025No Verifiable Reward for Prosody: Toward Preference-Guided Prosody Learning in TTScited by 4
  10. 2025UniSE: A Unified Framework for Decoder-Only Autoregressive LM-Based Speech Enhancementcited by 4
  11. 2026Beyond Fixed Frames: Dynamic Character-Aligned Speech Tokenizationcited by 4
  12. 2026dots.tts Technical Reportcited by 4
  13. 2025TTS-1 Technical Reportcited by 3
  14. 2025Inference-time Scaling for Diffusion-based Audio Super-resolutioncited by 3
  15. 2025SenSE: Semantic-Aware High-Fidelity Universal Speech Enhancementcited by 3
  16. 2025MOSS-Speech: Towards True Speech-to-Speech Models Without Text Guidancecited by 3
  17. 2025Multi-Reward GRPO for Stable and Prosodic Single-Codebook TTS LLMs at Scalecited by 3
  18. 2026Sparse Autoencoders for Interpretable Emotion Control in Text-to-Speechcited by 3
  19. 2025Shallow Flow Matching for Coarse-to-Fine Text-to-Speech Synthesiscited by 2
  20. 2025UniTTS: An end-to-end TTS system without decoupling of acoustic and semantic informationcited by 2
  21. 2025MoE-TTS: Enhancing Out-of-Domain Text Understanding for Description-based TTS via Mixture-of-Expertscited by 2
  22. 2025Finite Scalar Quantization Enables Redundant and Transmission-Robust Neural Audio Compression at Low Bit-ratescited by 2
  23. 2025High-Fidelity Speech Enhancement via Discrete Audio Tokenscited by 2
  24. 2025Principled Coarse-Grained Acceptance for Speculative Decoding in Speechcited by 2
  25. 2026UniSRCodec: Unified and Low-Bitrate Single Codebook Codec with Sub-Band Reconstructioncited by 2
  26. 2026Scaling Speech Tokenizers with Diffusion Autoencoderscited by 2
  27. 2026Reconstruct! Don't Encode: Self-Supervised Representation Reconstruction Loss for High-Intelligibility and Low-Latency Streaming Neural Audio Codeccited by 2
  28. 2026MOSS-TTSD: Text to Spoken Dialogue Generationcited by 2
  29. 2025Efficient Speech Language Modeling via Energy Distance in Continuous Latent Spacecited by 1
  30. 2025Eliminating stability hallucinations in llm-based tts models via attention guidancecited by 1
  31. 2025U-Codec: Ultra Low Frame-rate Neural Speech Codec for Fast High-fidelity Speech Generationcited by 1
  32. 2026DeepASMR: LLM-Based Zero-Shot ASMR Speech Generation for Anyone of Any Voicecited by 1
  33. 2026MamTra: A Hybrid Mamba-Transformer Backbone for Speech Synthesiscited by 1
  34. 2026WavTTS: Towards High-Quality Zero-Shot TTS via Direct Raw Waveform Modelingcited by 1
  35. 2026FlashTTS: Fast Streaming TTS with MTP Acceleration and X-pred Mean Flow Distillationcited by 1
  36. 2026DELTA-TTS: Adapting Autoregressive Model into Diffusion Language Model for Text-to-Speechcited by 1
  37. 2026Language-Statistical Analysis of Neural Audio Codec Tokens Across Architectures, Corpora, and Noise Conditionscited by 1
  38. 2025GOAT-TTS: Expressive and Realistic Speech Generation via A Dual-Branch LLMcited by 0

50 most cited of 70 All 70 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Llama-Mimi: Exploring the Limits of Flattened Speech Language Modeling
  2. 2023Speech Translation with Large Language Models: An Industrial Practice
  3. 2024Get Large Language Models Ready to Speak: A Late-fusion Approach for Speech Generation
  4. 2026LLaDA-TTS: Unifying Speech Synthesis and Zero-Shot Editing via Masked Diffusion Modeling
  5. 2024Low Frame-rate Speech Codec: a Codec Designed for Fast High-quality Speech LLM Training and Inference
  6. 2023A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speech
  7. 2025Scaling Auditory Cognition via Test-Time Compute in Audio Language Models
  8. 2024CM-TTS: Enhancing Real Time Text-to-Speech Synthesis Efficiency through Weighted Samplers and Consistency Models