# General Intelligence Requires Reward-based Pretraining

arXiv: https://arxiv.org/abs/2502.19402 · HTML: https://mlsanity.com/paper/2502.19402/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 75 references that are in the map.
- First posted on arXiv 26 Feb 2025.

## Builds on (most cited first, 30 of 50)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations](https://mlsanity.com/paper/2312.08935/) (2023)
- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://mlsanity.com/paper/1712.01815/) (2017)
- [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014)
- [GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models](https://mlsanity.com/paper/2410.05229/) (2024)
- [DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition](https://mlsanity.com/paper/1310.1531/) (2013)
- [A Deep Reinforced Model for Abstractive Summarization](https://mlsanity.com/paper/1705.04304/) (2017)
- [Large Language Models Are Reasoning Teachers](https://mlsanity.com/paper/2212.10071/) (2022)
- [Data Distributional Properties Drive Emergent In-Context Learning in Transformers](https://mlsanity.com/paper/2205.05055/) (2022)
- [Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking](https://mlsanity.com/paper/2403.09629/) (2024)
- [Can language models learn from explanations in context?](https://mlsanity.com/paper/2204.02329/) (2022)
- [Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks](https://mlsanity.com/paper/2307.02477/) (2023)
- [ReFT: Reasoning with Reinforced Fine-Tuning](https://mlsanity.com/paper/2401.08967/) (2024)
- [V-STaR: Training Verifiers for Self-Taught Reasoners](https://mlsanity.com/paper/2402.06457/) (2024)
- [A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends](https://mlsanity.com/paper/2301.05712/) (2023)
- [Pretrained Transformers as Universal Computation Engines](https://mlsanity.com/paper/2103.05247/) (2021)
- [Human Pose Estimation with Iterative Error Feedback](https://mlsanity.com/paper/1507.06550/) (2015)
- [On Memorization of Large Language Models in Logical Reasoning](https://mlsanity.com/paper/2410.23123/) (2024)
- [Learning to See by Moving](https://mlsanity.com/paper/1505.01596/) (2015)
- [Learning to See by Looking at Noise](https://mlsanity.com/paper/2106.05963/) (2021)

## Built on by (most cited first, 12 of 12)

- [RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs](https://mlsanity.com/paper/2509.21128/) (2025)
- [RLP: Reinforcement as a Pretraining Objective](https://mlsanity.com/paper/2510.01265/) (2025)
- [Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis](https://mlsanity.com/paper/2505.11581/) (2025)
- [Training Language Models via Neural Cellular Automata](https://mlsanity.com/paper/2603.10055/) (2026)
- [Reinforcement Learning on Pre-Training Data](https://mlsanity.com/paper/2509.19249/) (2025)
- [Procedural Pretraining: Warming Up Language Models with Abstract Data](https://mlsanity.com/paper/2601.21725/) (2026)
- [Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning](https://mlsanity.com/paper/2505.22308/) (2025)
- [Learning from Synthetic Data Improves Multi-hop Reasoning](https://mlsanity.com/paper/2603.02091/) (2026)
- [Learning from Mistakes: Negative Reasoning Samples Enhance Out-of-Domain Generalization](https://mlsanity.com/paper/2601.04992/) (2026)
- [Incorporating Self-Rewriting into Large Language Model Reasoning Reinforcement](https://mlsanity.com/paper/2511.16331/) (2025)
- [Syntax Without Semantics: Teaching Large Language Models to Code in an Unseen Language](https://mlsanity.com/paper/2605.15607/) (2026)
- [An Investigation of the NeurIPS and ICML 2025 Position Tracks](https://mlsanity.com/paper/2608.16894/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated impressive real-world utility, exemplifying artificial useful intelligence (AUI). However, their ability to reason adaptively and robustly -- the hallmarks of artificial general intelligence (AGI) -- remains fragile. While LLMs seemingly succeed in commonsense reasoning, programming, and mathematics, they struggle to generalize algorithmic understanding across novel contexts. Our experiments with algorithmic tasks in esoteric programming languages reveal that LLM's reasoning overfits to the training data and is limited in its transferability. We hypothesize that the core issue underlying such limited transferability is the coupling of reasoning and knowledge in LLMs. To transition from AUI to AGI, we propose disentangling knowledge and reasoning through three key directions: (1) pretaining to reason using RL from scratch as an alternative to the widely used next-token prediction pretraining, (2) using a curriculum of synthetic tasks to ease the learning of a reasoning prior for RL that can then be transferred to natural language tasks, and (3) learning more generalizable reasoning functions using a small context window to reduce exploiting spurious correlations between tokens. Such a reasoning system coupled with a trained retrieval system and a large external memory bank as a knowledge store can overcome several limitations of existing architectures at learning to reason in novel scenarios.