# Universal Reinforcement Learning Algorithms: Survey and Experiments

arXiv: https://arxiv.org/abs/1705.10557 · HTML: https://mlsanity.com/paper/1705.10557/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 May 2017.

## Built on by (most cited first, 4 of 4)

- [Offline Policy Comparison under Limited Historical Agent-Environment Interactions](https://mlsanity.com/paper/2106.03934/) (2021)
- [Computing Complexity-aware Plans Using Kolmogorov Complexity](https://mlsanity.com/paper/2109.10303/) (2021)
- [Data-Centric Approach to Constrained Machine Learning: A Case Study on Conway's Game of Life](https://mlsanity.com/paper/2408.12778/) (2024)
- [Golden Handcuffs make safer AI agents](https://mlsanity.com/paper/2604.13609/) (2026)

## Abstract (arXiv)

Many state-of-the-art reinforcement learning (RL) algorithms typically assume that the environment is an ergodic Markov Decision Process (MDP). In contrast, the field of universal reinforcement learning (URL) is concerned with algorithms that make as few assumptions as possible about the environment. The universal Bayesian agent AIXI and a family of related URL algorithms have been developed in this setting. While numerous theoretical optimality results have been proven for these agents, there has been no empirical investigation of their behavior to date. We present a short and accessible survey of these URL algorithms under a unified notation and framework, along with results of some experiments that qualitatively illustrate some properties of the resulting policies, and their relative performance on partially-observable gridworld environments. We also present an open-source reference implementation of the algorithms which we hope will facilitate further understanding of, and experimentation with, these ideas.