mlsanityData to 14 Oct 2026

NRGPT: An Energy-based Alternative for GPT

arXiv 2512.16762 ↗First posted 18 Dec 2025cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Generative Pre-trained Transformer (GPT) architectures are the most popular design for language modeling. Energy-based modeling is a different paradigm that views inference as a dynamical process operating on an energy landscape. We propose a minimal modification of the GPT setting to unify it with the EBM framework. The inference step of our model, which we call eNeRgy-GPT (NRGPT), is conceptualized as an exploration of the tokens on the energy landscape. We prove, and verify empirically, that under certain circumstances this exploration becomes gradient descent, although they don't necessarily lead to the best performing models. We demonstrate that our model performs well for simple language (Shakespeare dataset), algebraic ListOPS tasks, and richer settings such as OpenWebText language modeling. We also observe that our models may be more resistant to overfitting, doing so only during very long training.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 20 of its 30 references that are in the map. First posted on arXiv 18 Dec 2025.

Builds on

20 of 30 references.

References · 20

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2023Qwen Technical Reportcited by 3,344
  9. 2019Root Mean Square Layer Normalizationcited by 1,480
  10. 2022Transformers learn in-context by gradient descentcited by 732
  11. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  12. 2016Dense Associative Memory for Pattern Recognitioncited by 236
8 more
  1. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 178
  2. 2023A mathematical perspective on Transformerscited by 164
  3. 2023The emergence of clusters in self-attention dynamicscited by 140
  4. 2018ListOps: A Diagnostic Dataset for Latent Tree Learningcited by 129
  5. 2023Energy Transformercited by 71
  6. 2023Simplifying Transformer Blockscited by 37
  7. 2025Energy-Based Transformers are Scalable Learners and Thinkerscited by 31
  8. 2022Transformers from an Optimization Perspectivecited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Energy-Based Dynamical Models for Neurocomputation, Learning, and Optimizationcited by 1
  2. 2026Revisiting Transformer Layer Parameterization Through Causal Energy Minimizationcited by 0
  3. 2026Hyperparameter Transfer for Dense Associative Memoriescited by 0
  4. 2026Continual Learning in Modern Hopfield Networks with an Application to Diffusion Modelscited by 0
  5. 2026When Context Misleads: Surprisal, Energy and Attention Entropy as Metrics of Coherence Illusions in LLMscited by 0
  6. 2026Energy-Based Transformers as Predictors of Reading Difficultycited by 0
  7. 2026An all-transistor physical-learning transformer can be trained to perform in-context learningcited by 0
  8. 2026Test-Time Forecasting: Recurrent Depth Enables In-Context Dynamics Predictioncited by 0