# Representation-Driven Reinforcement Learning

arXiv: https://arxiv.org/abs/2305.19922 · HTML: https://mlsanity.com/paper/2305.19922/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 54 references that are in the map.
- First posted on arXiv 31 May 2023.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [Successor Features for Transfer in Reinforcement Learning](https://mlsanity.com/paper/1606.05312/) (2016)
- [Visual Reinforcement Learning with Imagined Goals](https://mlsanity.com/paper/1807.04742/) (2018)
- [Beyond UCB: Optimal and Efficient Contextual Bandits with Regression Oracles](https://mlsanity.com/paper/2002.04926/) (2020)
- [Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling](https://mlsanity.com/paper/1802.09127/) (2018)
- [Equivariant Architectures for Learning in Deep Weight Spaces](https://mlsanity.com/paper/2301.12780/) (2023)
- [Evolution-Guided Policy Gradient in Reinforcement Learning](https://mlsanity.com/paper/1805.07917/) (2018)
- [Simple random search provides a competitive approach to reinforcement learning](https://mlsanity.com/paper/1803.07055/) (2018)
- [Learning Action Representations for Reinforcement Learning](https://mlsanity.com/paper/1902.00183/) (2019)
- [Neural Contextual Bandits with Deep Representation and Shallow Exploration](https://mlsanity.com/paper/2012.01780/) (2020)
- [Neural Contextual Bandits without Regret](https://mlsanity.com/paper/2107.03144/) (2021)
- [The Natural Language of Actions](https://mlsanity.com/paper/1902.01119/) (2019)
- [Distributional Policy Optimization: An Alternative Approach for Continuous Control](https://mlsanity.com/paper/1905.09855/) (2019)
- [Online Limited Memory Neural-Linear Bandits with Likelihood Matching](https://mlsanity.com/paper/2102.03799/) (2021)
- [ERL-Re2: Efficient Evolutionary Reinforcement Learning with Shared State Representation and Individual Policy Representation](https://mlsanity.com/paper/2210.17375/) (2022)
- [Uncertainty Estimation Using Riemannian Model Dynamics for Offline Reinforcement Learning](https://mlsanity.com/paper/2102.11327/) (2021)
- [Language is Power: Representing States Using Natural Language in Reinforcement Learning](https://mlsanity.com/paper/1910.02789/) (2019)

## Built on by (most cited first, 4 of 4)

- [Embedding-Aligned Language Models](https://mlsanity.com/paper/2406.00024/) (2024)
- [Reinforcement Learning with Discrete Diffusion Policies for Combinatorial Action Spaces](https://mlsanity.com/paper/2509.22963/) (2025)
- [Spectral Bellman Method: Unifying Representation and Exploration in RL](https://mlsanity.com/paper/2507.13181/) (2025)
- [Distributions as Actions: A Unified Framework for Diverse Action Spaces](https://mlsanity.com/paper/2506.16608/) (2025)

## Abstract (arXiv)

We present a representation-driven framework for reinforcement learning. By representing policies as estimates of their expected values, we leverage techniques from contextual bandits to guide exploration and exploitation. Particularly, embedding a policy network into a linear feature space allows us to reframe the exploration-exploitation problem as a representation-exploitation problem, where good policy representations enable optimal exploration. We demonstrate the effectiveness of this framework through its application to evolutionary and policy gradient-based approaches, leading to significantly improved performance compared to traditional methods. Our framework provides a new perspective on reinforcement learning, highlighting the importance of policy representation in determining optimal exploration-exploitation strategies.