mlsanityData to 14 Oct 2026

Autoregressive Language Models are Secretly Energy-Based Models: Insights into the Lookahead Capabilities of Next-Token Prediction

arXiv 2512.15605 ↗First posted 17 Dec 2025cs.LG stat.MLcited by 8

Not on the map. Why?

Abstract (arXiv)

Autoregressive models (ARMs) currently constitute the dominant paradigm for large language models (LLMs). Energy-based models (EBMs) represent another class of models, which have historically been less prevalent in LLM development, yet naturally characterize the optimal policy in post-training alignment. In this paper, we provide a unified view of these two model classes. Taking the chain rule of probability as a starting point, we establish an explicit bijection between ARMs and EBMs in function space, which we show to correspond to a special case of the soft Bellman equation in maximum entropy reinforcement learning. Building upon this bijection, we derive the equivalence between supervised learning of ARMs and EBMs. Furthermore, we analyze the distillation of EBMs into ARMs by providing theoretical error bounds. Our results provide insights into the ability of ARMs to plan ahead, despite being based on the next-token prediction paradigm.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 35 of its 51 references that are in the map. First posted on arXiv 17 Dec 2025.

Builds on

35 of 51 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  5. 2020Learning to summarize from human feedbackcited by 2,588
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  7. 2016Generative Adversarial Imitation Learningcited by 1,597
  8. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  9. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  10. 2021Flow Network based Generative Models for Non-Iterative Diverse Candidate Generationcited by 392
  11. 2021GFlowNet Foundationscited by 290
  12. 2019A Theory of Regularized Markov Decision Processescited by 286
23 more
  1. 2021How to Train Your Energy-Based Modelscited by 274
  2. 2022Trajectory balance: Improved credit assignment in GFlowNetscited by 251
  3. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 182
  4. 2017A unified view of entropy-regularized Markov decision processescited by 182
  5. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  6. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 173
  7. 2024The pitfalls of next-token predictioncited by 160
  8. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 135
  9. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 120
  10. 2023Amortizing intractable inference in large language modelscited by 97
  11. 2024Probabilistic Inference in Language Models via Twisted Sequential Monte Carlocited by 90
  12. 2024Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learningcited by 74
  13. 2023Generative Flow Networks as Entropy-Regularized RLcited by 64
  14. 2018Differentiable Dynamic Programming for Structured Prediction and Attentioncited by 58
  15. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48
  16. 2024The Elements of Differentiable Programmingcited by 43
  17. 2024Transformers are Universal In-context Learnerscited by 43
  18. 2025Loss Functions and Operators Generated by f-Divergencescited by 10
  19. 2024The Mystery of the Pathological Path-star Task for Language Modelscited by 8
  20. 2023Formal Aspects of Language Modelingcited by 7
  21. 2017A short variational proof of equivalence between policy gradients and soft Q learningcited by 3
  22. 2025Joint Learning of Energy-based Models and their Partition Functioncited by 3
  23. 2025ShiQ: Bringing back Bellman to LLMscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026LoopUS: Recasting Pretrained LLMs into Looped Latent Refinement Modelscited by 15
  2. 2026Ontology-Constrained Neural Reasoning in Enterprise Agentic Systems: A Neurosymbolic Architecture for Domain-Grounded AI Agentscited by 3
  3. 2026Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoningcited by 2
  4. 2026Graph Energy Matching: Transport-Aligned Energy-Based Modeling for Graph Generationcited by 1
  5. 2026Constitutional On-Policy Safe Distillationcited by 1
  6. 2026Large Language Models Can Take False First Steps at Inference-time Planningcited by 0
  7. 2026d-OPD: Future-Aware On-Policy Distillation for Block Diffusion Language Modelscited by 0
  8. 2026Revealing Epistemic Uncertainty in MLLMs via Causal-Invariant Maskingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ForkLeft: Entropy-First Rollouts for Prefix-Aligned Autoregressive-to-Diffusion Distillation
  2. 2024Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement
  3. 2024Beyond Autoregression: Fast LLMs via Self-Distillation Through Time
  4. 2025A Theoretical Lens for RL-Tuned Language Models via Energy-Based Models
  5. 2026LLMs as High-Dimensional Nonlinear Autoregressive Models with Attention: Training, Alignment and Inference
  6. 2026JustOPD: On-Policy Distillation from Autoregressive to Diffusion Language Models
  7. 2026ETS: Energy-Guided Test-Time Scaling for Training-Free RL Alignment
  8. 2026Energy Consumption Analysis of Discrete Diffusion and Autoregressive Language Models