mlsanityData to 14 Oct 2026

L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Models

arXiv 2505.17505 ↗First posted 23 May 2025cs.CLcited by 15

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have achieved notable progress. Despite their success, next-token prediction (NTP), the dominant method for LLM training and inference, is constrained in both contextual coverage and inference efficiency due to its inherently sequential process. To overcome these challenges, we propose leap multi-token prediction~(L-MTP), an innovative token prediction method that extends the capabilities of multi-token prediction (MTP) by introducing a leap-based mechanism. Unlike conventional MTP, which generates multiple tokens at adjacent positions, L-MTP strategically skips over intermediate tokens, predicting non-sequential ones in a single forward pass. This structured leap not only enhances the model's ability to capture long-range dependencies but also enables a decoding strategy specially optimized for non-sequential leap token generation, effectively accelerating inference. We theoretically demonstrate the benefit of L-MTP in improving inference efficiency. Experiments across diverse benchmarks validate its merit in boosting both LLM performance and inference speed. The source code is available at https://github.com/Xiaohao-Liu/L-MTP.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 76 of its 97 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

76 of 97 references.

References · 76

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  8. 2024Qwen2.5 Technical Reportcited by 4,942
  9. 2023Let's Verify Step by Stepcited by 3,789
  10. 2021Program Synthesis with Large Language Modelscited by 3,637
  11. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  12. 2025Gemma 3 Technical Reportcited by 1,824
38 more
  1. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  3. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  4. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  5. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  6. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  7. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  8. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  9. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  10. 2023Instruction Tuning with GPT-4cited by 686
  11. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  12. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  13. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  14. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 578
  15. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
  16. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 426
  17. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  18. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 365
  19. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  20. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 323
  21. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  22. 2025DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decompositioncited by 287
  23. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 269
  24. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 257
  25. 2025TokenSkip: Controllable Chain-of-Thought Compression in LLMscited by 249
  26. 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agentscited by 244
  27. 2020ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-trainingcited by 222
  28. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  29. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillationcited by 165
  30. 2024Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputscited by 150
  31. 2024Rho-1: Not All Tokens Are What You Needcited by 133
  32. 2023SpecTr: Fast Speculative Decoding via Optimal Transportcited by 129
  33. 2024Empowering Biomedical Discovery with AI Agentscited by 118
  34. 2024Hydra: Sequentially-Dependent Draft Heads for Medusa Decodingcited by 117
  35. 2024Data-efficient Fine-tuning for LLM-based Recommendationcited by 104
  36. 2024Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuningcited by 96
  37. 2023HyperAttention: Long-context Attention in Near-Linear Timecited by 85
  38. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 12.

15 papers.

Sort:

By area:

  1. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 30
  2. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matchingcited by 21
  3. 2025Beyond Multi-Token Prediction: Pretraining LLMs with Future Summariescited by 15
  4. 2026BATQuant: Outlier-resilient MXFP4 Quantization via Learnable Block-wise Optimizationcited by 6
  5. 2026FreeAct: Freeing Activations for LLM Quantizationcited by 4
  6. 2026NITP: Next Implicit Token Prediction for LLM Pre-trainingcited by 4
  7. 2026Know You Before You Speak: User-State Modeling for LLM Personalization in Multi-Turn Conversationcited by 3
  8. 2026Lingua-SafetyBench: A Benchmark for Safety Evaluation of Multilingual Vision-Language Modelscited by 2
  9. 2026SSD: Spatially Speculative Decoding Accelerates Autoregressive Image Generationcited by 2
  10. 2026Temporal Guidance for Large Language Modelscited by 0
  11. 2026How Transformers Learn to Plan via Multi-Token Predictioncited by 0
  12. 2026Enhancing Decision-Making with Large Language Models through Multi-Agent Fictitious Playcited by 0
3 more
  1. 2026HyperZip: Efficient Data Compression through Personalized Diffusion LLMs with Hypernetworkscited by 0
  2. 2026SEED: Self-Speculative Decoding via Implicit Encoder-Decodercited by 0
  3. 2026ReGRA: Residual-Gradient Adaptation for Complementary Low-Rank Fine-Tuningcited by 0