mlsanityData to 14 Oct 2026

Faster Language Models with Better Multi-Token Prediction Using Tensor Decomposition

arXiv 2410.17765 ↗First posted 23 Oct 2024cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

We propose a new model for multi-token prediction in transformers, aiming to enhance sampling efficiency without compromising accuracy. Motivated by recent work that predicts the probabilities of subsequent tokens using multiple heads, we connect this approach to rank-1 canonical tensor decomposition. By generalizing it to a rank-r canonical probability decomposition, we develop an improved model that predicts multiple tokens simultaneously. This model can also be interpreted as a mixture of experts, allowing us to leverage successful techniques from that domain for efficient and robust training. Importantly, the overall overhead for training and sampling remains low. Our method demonstrates significant improvements in inference speed for both text and code generation tasks, proving particularly beneficial within the self-speculative decoding paradigm. It maintains its effectiveness across various model sizes and training epochs, highlighting its robustness and scalability.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 25 of its 33 references that are in the map. First posted on arXiv 23 Oct 2024.

Builds on

25 of 33 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  4. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  5. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  6. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  7. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  8. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 578
  9. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  10. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 365
  11. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  12. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 323
13 more
  1. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
  2. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 298
  3. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  4. 2023Accelerating Transformer Inference for Translation via Parallel Decodingcited by 124
  5. 2022CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generationcited by 87
  6. 2021A Practical Survey on Faster and Lighter Transformerscited by 73
  7. 2024OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structurecited by 67
  8. 2020Accelerating Feedforward Computation via Parallel Nonlinear Equation Solvingcited by 52
  9. 2023SPEED: Speculative Pipelined Execution for Efficient Decodingcited by 47
  10. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Modelscited by 42
  11. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Modelscited by 38
  12. 2024Multilinear Mixture of Experts: Scalable Expert Specialization through Factorizationcited by 28
  13. 2024DySpec: Faster Speculative Decoding with Dynamic Token Tree Structurecited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 6.

8 papers.

  1. 2026Breaking the Factorization Barrier in Diffusion Language Modelscited by 11
  2. 2025Fast and Expressive Multi-Byte Prediction with Probabilistic Circuitscited by 9
  3. 2026Tensorizing Engram: Sharing Latents Across N-Gram Embeddings is Beneficial in LLMscited by 3
  4. 2026Tensor-Train Joint Modeling for Few-Step Discrete Diffusioncited by 1
  5. 2026DBLAST: Dependent Block Drafting for Stochastic Speculative Decodingcited by 1
  6. 2025Entering the Era of Discrete Diffusion Models: A Benchmark for Schr"odinger Bridges and Entropic Optimal Transportcited by 0
  7. 2026AdaMTP: An Adaptive Training Paradigm for Multi-Token Predictioncited by 0
  8. 2026LightMTP: Lightweight Latent Multi-Token Predictioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Direct Multi-Token Decoding
  2. 2025Parallel Token Prediction for Language Models
  3. 2026Multi-Token Prediction via Self-Distillation
  4. 2025Pre-Training Curriculum for Multi-Token Prediction in Language Models
  5. 2024Optimized Multi-Token Joint Decoding with Auxiliary Model for LLM Inference
  6. 2026Distribution-Aware Companding Quantization of Large Language Models
  7. 2024Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recycling
  8. 2026Pair-In, Pair-Out: Latent Multi-Token Prediction for Efficient LLMs