mlsanityData to 7 Oct 2026

Cognitively Inspired Energy-Based World Models

arXiv 2406.08862 ↗First posted 13 Jun 2024cs.LGcited by 1

On the map: World models → World models for language agents

Abstract (arXiv)

One of the predominant methods for training world models is autoregressive prediction in the output space of the next element of a sequence. In Natural Language Processing (NLP), this takes the form of Large Language Models (LLMs) predicting the next token; in Computer Vision (CV), this takes the form of autoregressive models predicting the next frame/token/pixel. However, this approach differs from human cognition in several respects. First, human predictions about the future actively influence internal cognitive processes. Second, humans naturally evaluate the plausibility of predictions regarding future states. Based on this capability, and third, by assessing when predictions are sufficient, humans allocate a dynamic amount of time to make a prediction. This adaptive process is analogous to System 2 thinking in psychology. All these capabilities are fundamental to the success of humans at high-level reasoning and planning. Therefore, to address the limitations of traditional autoregressive models lacking these human-like capabilities, we introduce Energy-Based World Models (EBWM). EBWM involves training an Energy-Based Model (EBM) to predict the compatibility of a given context and a predicted future state. In doing so, EBWM enables models to achieve all three facets of human cognition described. Moreover, we developed a variant of the traditional autoregressive transformer tailored for Energy-Based models, termed the Energy-Based Transformer (EBT). Our results demonstrate that EBWM scales better with data and GPU Hours than traditional autoregressive transformers in CV, and that EBWM offers promising early scaling in NLP. Consequently, this approach offers an exciting path toward training future models capable of System 2 thinking and intelligently searching across state spaces.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 54 of its 91 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

54 of 91 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  11. 2017Neural Discrete Representation Learningcited by 4,650
  12. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
38 more
  1. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  2. 2023Mistral 7Bcited by 3,065
  3. 2017The Kinetics Human Action Video Datasetcited by 1,894
  4. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  5. 2018World Modelscited by 1,444
  6. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  7. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  8. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  9. 2023Textbooks Are All You Needcited by 561
  10. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  11. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  12. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  13. 2021How to Train Your Energy-Based Modelscited by 271
  14. 2023Think before you speak: Training Language Models With Pause Tokenscited by 254
  15. 2023Sequential Modeling Enables Scalable Learning for Large Vision Modelscited by 212
  16. 2019Scaling Autoregressive Video Modelscited by 189
  17. 2021Direct speech-to-speech translation with discrete unitscited by 148
  18. 2022TransDreamer: Reinforcement Learning with Transformer World Modelscited by 143
  19. 2022Diffusion Models for Video Prediction and Infillingcited by 134
  20. 2020Improved Contrastive Divergence Training of Energy Based Modelscited by 127
  21. 2023The Generative AI Paradox: "What It Can Create, It May Not Understand"cited by 97
  22. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Surveycited by 82
  23. 2021Unsupervised Learning of Compositional Energy Conceptscited by 71
  24. 2023Energy Transformercited by 69
  25. 2024LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancementcited by 69
  26. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  27. 2020Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesiscited by 54
  28. 2024Uncertainty-Based Abstention in LLMs Improves Safety and Reduces Hallucinationscited by 51
  29. 2020Energy-Based Reranking: Improving Neural Machine Translation Using Energy-Based Modelscited by 44
  30. 2024A Survey on Long Video Generation: Challenges, Methods, and Prospectscited by 40
  31. 2023Transformers in Speech Processing: A Surveycited by 38
  32. 2023World Models and Predictive Coding for Cognitive and Developmental Robotics: Frontiers and Challengescited by 35
  33. 2020HAMLET: A Hierarchical Multimodal Attention-based Human Activity Recognition Algorithmcited by 29
  34. 2023A Survey of Diffusion Models in Natural Language Processingcited by 16
  35. 2022Representation Learning in Deep RL via Discrete Information Bottleneckcited by 13
  36. 2023Neural Foundations of Mental Simulation: Future Prediction of Latent Representations on Dynamic Scenescited by 13
  37. 2023DiffAR: Denoising Diffusion Autoregressive Model for Raw Speech Waveform Generationcited by 11
  38. 2024Exploring the Limitations of Large Language Models in Compositional Relation Reasoningcited by 9

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Three-in-One World Model: Energy-Based Consistency, Prediction, and Counterfactual Inference for Marketing Interventioncited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation
  2. 2024CogDPM: Diffusion Probabilistic Models via Cognitive Predictive Coding
  3. 2024Learning World Models With Hierarchical Temporal Abstractions: A Probabilistic Perspective
  4. 2026Metacognitive Reasoning in Energy Based Models using Instance Based Learning Theory
  5. 2025World Models in Artificial Intelligence: Sensing, Learning, and Reasoning Like a Child
  6. 2026World Models Meet Language Models: On the Complementarity of Concrete and Abstract Reasoning
  7. 2023Visual cognition in multimodal large language models
  8. 2026X-Foresight: A Joint Vision-Action Causal Forecasting Network via Predictive World Modeling