# The Missing Reward: Active Inference in the Era of Experience

arXiv: https://arxiv.org/abs/2508.05619 · HTML: https://mlsanity.com/paper/2508.05619/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 19 of its 42 references that are in the map.
- First posted on arXiv 7 Aug 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)

## Builds on (most cited first, 19 of 19)

- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [Carbon Emissions and Large Neural Network Training](https://mlsanity.com/paper/2104.10350/) (2021)
- [Faith and Fate: Limits of Transformers on Compositionality](https://mlsanity.com/paper/2305.18654/) (2023)
- [Transformers Can Do Bayesian Inference](https://mlsanity.com/paper/2112.10510/) (2021)
- [Emergent Analogical Reasoning in Large Language Models](https://mlsanity.com/paper/2212.09196/) (2022)
- [Will we run out of data? Limits of LLM scaling based on human-generated data](https://mlsanity.com/paper/2211.04325/) (2022)
- [iGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenes](https://mlsanity.com/paper/2012.02924/) (2020)
- [Large Language Models as Analogical Reasoners](https://mlsanity.com/paper/2310.01714/) (2023)
- [LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAI's o1 on PlanBench](https://mlsanity.com/paper/2409.13373/) (2024)
- [The free energy principle for action and perception: A mathematical review](https://mlsanity.com/paper/1705.09156/) (2017)
- [Deep Active Inference as Variational Policy Gradients](https://mlsanity.com/paper/1907.03876/) (2019)
- [Reinforcement Learning through Active Inference](https://mlsanity.com/paper/2002.12636/) (2020)
- [Deep Active Inference](https://mlsanity.com/paper/1709.02341/) (2017)
- [A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk](https://mlsanity.com/paper/1712.05796/) (2017)
- [Evaluating the Robustness of Analogical Reasoning in Large Language Models](https://mlsanity.com/paper/2411.14215/) (2024)
- [Learning Perception and Planning with Deep Active Inference](https://mlsanity.com/paper/2001.11841/) (2020)

## Built on by (most cited first, 2 of 2)

- [Architecting Trust in Artificial Epistemic Agents](https://mlsanity.com/paper/2603.02960/) (2026)
- [Intuition emerges in Maximum Caliber models at criticality](https://mlsanity.com/paper/2508.06477/) (2025)

## Abstract (arXiv)

This paper argues that Active Inference (AIF) provides a crucial foundation for developing autonomous AI agents capable of learning from experience without continuous human reward engineering. As AI systems begin to exhaust high-quality training data and rely on increasingly large human workforces for reward design, the current paradigm faces significant scalability challenges that could impede progress toward genuinely autonomous intelligence. The proposal for an ``Era of Experience,'' where agents learn from self-generated data, is a promising step forward. However, this vision still depends on extensive human engineering of reward functions, effectively shifting the bottleneck from data curation to reward curation. This highlights what we identify as the grounded-agency gap: the inability of contemporary AI systems to autonomously formulate, adapt, and pursue objectives in response to changing circumstances. We propose that AIF can bridge this gap by replacing external reward signals with an intrinsic drive to minimize free energy, allowing agents to naturally balance exploration and exploitation through a unified Bayesian objective. By integrating Large Language Models as generative world models with AIF's principled decision-making framework, we can create agents that learn efficiently from experience while remaining aligned with human values. This synthesis offers a compelling path toward AI systems that can develop autonomously while adhering to both computational and physical constraints.