# Extending Environments To Measure Self-Reflection In Reinforcement Learning

arXiv: https://arxiv.org/abs/2110.06890 · HTML: https://mlsanity.com/paper/2110.06890/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 46 references that are in the map.
- First posted on arXiv 13 Oct 2021.

## Builds on (most cited first, 11 of 11)

- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Leveraging Procedural Generation to Benchmark Reinforcement Learning](https://mlsanity.com/paper/1912.01588/) (2019)
- [Risks from Learned Optimization in Advanced Machine Learning Systems](https://mlsanity.com/paper/1906.01820/) (2019)
- [Gotta Learn Fast: A New Benchmark for Generalization in RL](https://mlsanity.com/paper/1804.03720/) (2018)
- [Bad Universal Priors and Notions of Optimality](https://mlsanity.com/paper/1510.04931/) (2015)
- [The Animal-AI Environment: Training and Testing Animal-Like Artificial Cognition](https://mlsanity.com/paper/1909.07483/) (2019)
- [Intelligence via ultrafilters: structural properties of some intelligence comparators of deterministic Legg-Hutter agents](https://mlsanity.com/paper/1910.09721/) (2019)
- [Reward-Punishment Symmetric Universal Intelligence](https://mlsanity.com/paper/2110.02450/) (2021)
- [Detecting Qualia in Natural and Artificial Agents](https://mlsanity.com/paper/1712.04020/) (2017)
- [Introspective Agents: Confidence Measures for General Value Functions](https://mlsanity.com/paper/1606.05593/) (2016)

## Built on by (most cited first, 3 of 3)

- [On the Computation of Meaning, Language Models and Incomprehensible Horrors](https://mlsanity.com/paper/2304.12686/) (2023)
- [Why Is Anything Conscious?](https://mlsanity.com/paper/2409.14545/) (2024)
- [Representation and Invariance in Reinforcement Learning](https://mlsanity.com/paper/2112.07752/) (2021)

## Abstract (arXiv)

We consider an extended notion of reinforcement learning in which the environment can simulate the agent and base its outputs on the agent's hypothetical behavior. Since good performance usually requires paying attention to whatever things the environment's outputs are based on, we argue that for an agent to achieve on-average good performance across many such extended environments, it is necessary for the agent to self-reflect. Thus weighted-average performance over the space of all suitably well-behaved extended environments could be considered a way of measuring how self-reflective an agent is. We give examples of extended environments and introduce a simple transformation which experimentally seems to increase some standard RL agents' performance in a certain type of extended environment.