mlsanityData to 14 Oct 2026

Alternatives To Next Token Prediction In Text Generation -- A Survey

arXiv 2509.24435 ↗First posted 29 Sep 2025cs.CL cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

The paradigm of Next Token Prediction (NTP) has driven the unprecedented success of Large Language Models (LLMs), but is also the source of their most persistent weaknesses such as poor long-term planning, error accumulation, and computational inefficiency. Acknowledging the growing interest in exploring alternatives to NTP, the survey describes the emerging ecosystem of alternatives to NTP. We categorise these approaches into five main families: (1) Multi-Token Prediction, which targets a block of future tokens instead of a single one; (2) Plan-then-Generate, where a global, high-level plan is created upfront to guide token-level decoding; (3) Latent Reasoning, which shifts the autoregressive process itself into a continuous latent space; (4) Continuous Generation Approaches, which replace sequential generation with iterative, parallel refinement through diffusion, flow matching, or energy-based methods; and (5) Non-Transformer Architectures, which sidestep NTP through their inherent model structure. By synthesizing insights across these methods, this survey offers a taxonomy to guide research into models that address the known limitations of token-level generation to develop new transformative models for natural language processing.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 63 of its 73 references that are in the map. First posted on arXiv 29 Sep 2025.

Builds on

63 of 73 references.

References · 63

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  3. 2020Longformer: The Long-Document Transformercited by 3,413
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  5. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  6. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  7. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  8. 2023Reasoning with Language Model is Planning with World Modelcited by 861
  9. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
  10. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 797
  11. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  12. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
38 more
  1. 2024A Survey on Large Language Models for Code Generationcited by 658
  2. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 531
  3. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 501
  4. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 415
  5. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 409
  6. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  7. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 235
  8. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 235
  9. 2020ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-trainingcited by 222
  10. 2024Scaling up Masked Diffusion Models on Textcited by 199
  11. 2024The pitfalls of next-token predictioncited by 160
  12. 2024Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planningcited by 135
  13. 2023Accelerating Transformer Inference for Translation via Parallel Decodingcited by 124
  14. 2023Future Lens: Anticipating Subsequent Tokens from a Single Hidden Statecited by 121
  15. 2023Flow Matching for Scalable Simulation-Based Inferencecited by 101
  16. 2020Residual Energy-Based Models for Text Generationcited by 95
  17. 2024Energy-Based Diffusion Language Models for Text Generationcited by 89
  18. 2024Large Concept Models: Language Modeling in a Sentence Representation Spacecited by 89
  19. 2024Variational Flow Matching for Graph Generationcited by 83
  20. 2025PlanGenLLMs: A Modern Survey of LLM Planning Capabilitiescited by 83
  21. 2022A General Survey on Attention Mechanisms in Deep Learningcited by 64
  22. 2023PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Modelcited by 58
  23. 2021Plan-then-Generate: Controlled Data-to-Text Generation via Planningcited by 52
  24. 2024From Tokens to Words: On the Inner Lexicon of LLMscited by 48
  25. 2024Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Modelscited by 46
  26. 2023Diffusion Models for Non-autoregressive Text Generation: A Surveycited by 44
  27. 2024PlanRAG: A Plan-then-Retrieval Augmented Generation for Generative Large Language Models as Decision Makerscited by 42
  28. 2025LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilitiescited by 41
  29. 2024Why Do Large Language Models (LLMs) Struggle to Count Letters?cited by 38
  30. 2025LLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architecturescited by 38
  31. 2021Data-to-text Generation with Macro Planningcited by 34
  32. 2025Energy-Based Transformers are Scalable Learners and Thinkerscited by 31
  33. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  34. 2024TEncDM: Understanding the Properties of the Diffusion Model in the Space of Language Model Encodingscited by 22
  35. 2023A Survey of Diffusion Models in Natural Language Processingcited by 16
  36. 2024Retrieve-Plan-Generation: An Iterative Planning and Answering Framework for Knowledge-Intensive LLM Generationcited by 15
  37. 2025Computational-Statistical Tradeoffs at the Next-Token Prediction Barrier: Autoregressive and Imitation Learning under Misspecificationcited by 13
  38. 2024Learning to Plan for Language Modeling from Unlabeled Datacited by 12

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Toward Consistent World Models with Multi-Token Prediction and Latent Semantic Enhancementcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond Next-Token Prediction: A Performance Characterization of Diffusion versus Autoregressive Language Models
  2. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Models
  3. 2026AdaMTP: An Adaptive Training Paradigm for Multi-Token Prediction
  4. 2025Pre-Training Curriculum for Multi-Token Prediction in Language Models
  5. 2025On multi-token prediction for efficient LLM inference
  6. 2026NITP: Next Implicit Token Prediction for LLM Pre-training
  7. 2026Next Concept Prediction in Discrete Latent Space Leads to Stronger Language Models
  8. 2025Set Block Decoding is a Language Model Inference Accelerator