mlsanityData to 7 Oct 2026

Small-E: Small Language Model with Linear Attention for Efficient Speech Synthesis

arXiv 2406.04467 ↗First posted 6 Jun 2024eess.AS cs.CL cs.SDcited by 6

Not on the map. Why?

Abstract (arXiv)

Recent advancements in text-to-speech (TTS) powered by language models have showcased remarkable capabilities in achieving naturalness and zero-shot voice cloning. Notably, the decoder-only transformer is the prominent architecture in this domain. However, transformers face challenges stemming from their quadratic complexity in sequence length, impeding training on lengthy sequences and resource-constrained hardware. Moreover they lack specific inductive bias with regards to the monotonic nature of TTS alignments. In response, we propose to replace transformers with emerging recurrent architectures and introduce specialized cross-attention mechanisms for reducing repeating and skipping issues. Consequently our architecture can be efficiently trained on long samples and achieve state-of-the-art zero-shot voice cloning against baselines of comparable size. Our implementation and demos are available at https://github.com/theodorblackbird/lina-speech.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 29 of its 33 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

29 of 33 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  6. 2020Reformer: The Efficient Transformercited by 1,658
  7. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  8. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  9. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  10. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  11. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  12. 2022High Fidelity Neural Audio Compressioncited by 673
17 more
  1. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  2. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  3. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  4. 2023Simple and Controllable Music Generationcited by 480
  5. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  6. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  7. 2018Efficient Attention: Attention with Linear Complexitiescited by 366
  8. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
  9. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 233
  10. 2023Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesiscited by 150
  11. 2021An Attention Free Transformercited by 111
  12. 2023SoundStorm: Efficient Parallel Audio Generationcited by 111
  13. 2023Better speech synthesis through scalingcited by 63
  14. 2023VoiceFlow: Efficient Text-to-Speech with Rectified Flow Matchingcited by 49
  15. 2021One TTS Alignment To Rule Them Allcited by 47
  16. 2023Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Abilitycited by 16
  17. 2022Regotron: Regularizing the Tacotron2 architecture via monotonic alignment losscited by 3

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 3.

6 papers.

  1. 2024InkubaLM: A small language model for low-resource African languagescited by 18
  2. 2025Voice Cloning: Comprehensive Surveycited by 4
  3. 2025Gelina: Unified Speech and Gesture Synthesis via Interleaved Token Predictioncited by 2
  4. 2024Lina-Speech: Gated Linear Attention and Initial-State Tuning for Multi-Sample Prompting Text-To-Speech Synthesiscited by 1
  5. 2024Personal Intelligence System UniLM: Hybrid On-Device Small Language Model and Server-Based Large Language Model for Malay Nusantaracited by 0
  6. 2025Flamed-TTS: Flow Matching Attention-Free Models for Efficient Generating and Dynamic Pacing Zero-shot Text-to-Speechcited by 0