# From Parameters to Behaviors: Unsupervised Compression of the Policy Space

arXiv: https://arxiv.org/abs/2509.22566 · HTML: https://mlsanity.com/paper/2509.22566/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 32 of its 57 references that are in the map.
- First posted on arXiv 26 Sep 2025.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 30 of 32)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables](https://mlsanity.com/paper/1903.08254/) (2019)
- [On the Number of Linear Regions of Deep Neural Networks](https://mlsanity.com/paper/1402.1869/) (2014)
- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [On the Expressive Power of Deep Neural Networks](https://mlsanity.com/paper/1606.05336/) (2016)
- [Variational Intrinsic Control](https://mlsanity.com/paper/1611.07507/) (2016)
- [URLB: Unsupervised Reinforcement Learning Benchmark](https://mlsanity.com/paper/2110.15191/) (2021)
- [Fast Task Inference with Variational Intrinsic Successor Features](https://mlsanity.com/paper/1906.05030/) (2019)
- [Variational Option Discovery Algorithms](https://mlsanity.com/paper/1807.10299/) (2018)
- [Learning to Learn with Generative Models of Neural Network Checkpoints](https://mlsanity.com/paper/2209.12892/) (2022)
- [A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learning](https://mlsanity.com/paper/2208.07860/) (2022)
- [Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progress](https://mlsanity.com/paper/2206.01626/) (2022)
- [Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weights](https://mlsanity.com/paper/2209.14733/) (2022)
- [Policy Optimization via Importance Sampling](https://mlsanity.com/paper/1809.06098/) (2018)
- [Towards Scalable and Versatile Weight Space Learning](https://mlsanity.com/paper/2406.09997/) (2024)
- [Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimality](https://mlsanity.com/paper/2205.13521/) (2022)
- [Parameter-Based Value Functions](https://mlsanity.com/paper/2006.09226/) (2020)
- [Policy Manifold Search: Exploring the Manifold Hypothesis for Diversity-based Neuroevolution](https://mlsanity.com/paper/2104.13424/) (2021)
- [Generating Behaviorally Diverse Policies with Latent Diffusion Models](https://mlsanity.com/paper/2305.18738/) (2023)
- [Learning Optimal Deterministic Policies with Stochastic Policy Gradients](https://mlsanity.com/paper/2405.02235/) (2024)
- [Goal-Conditioned Generators of Deep Policies](https://mlsanity.com/paper/2207.01570/) (2022)
- [ClipUp: A Simple and Powerful Optimizer for Distribution-based Policy Evolution](https://mlsanity.com/paper/2008.02387/) (2020)
- [Reward-Free Policy Space Compression for Reinforcement Learning](https://mlsanity.com/paper/2202.11079/) (2022)
- [Towards Principled Unsupervised Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2502.08365/) (2025)
- [Enhancing Diversity in Parallel Agents: A Maximum State Entropy Exploration Story](https://mlsanity.com/paper/2505.01336/) (2025)

## Built on by (most cited first, 4 of 4)

- [Unsupervised Behavioral Compression: Learning Low-Dimensional Policy Manifolds through State-Occupancy Matching](https://mlsanity.com/paper/2603.27044/) (2026)
- [Representative Action Selection for Large Action Space: From Bandits to MDPs](https://mlsanity.com/paper/2511.22104/) (2025)
- [Learning in Low-Dimensional Subspaces: Orthogonal Bottlenecks for Reinforcement Learning](https://mlsanity.com/paper/2605.26012/) (2026)
- [Implicit Neural Representations of Individual Behavior](https://mlsanity.com/paper/2606.12200/) (2026)

## Abstract (arXiv)

Despite its recent successes, Deep Reinforcement Learning (DRL) is notoriously sample-inefficient. We argue that this inefficiency stems from the standard practice of optimizing policies directly in the high-dimensional and highly redundant parameter space Theta. This challenge is greatly compounded in multi-task settings. In this work, we develop a novel, unsupervised approach that compresses the policy parameter space Theta into a low-dimensional latent space Z. We train a generative model g:Z→Theta by optimizing a behavioral reconstruction loss, which ensures that the latent space is organized by functional similarity rather than proximity in parameterization. We conjecture that the inherent dimensionality of this manifold is a function of the environment's complexity, rather than the size of the policy network. We validate our approach in continuous control domains, showing that the parameterization of standard policy networks can be compressed up to five orders of magnitude while retaining most of its expressivity. As a byproduct, we show that the learned manifold enables task-specific adaptation via Policy Gradient operating in the latent space Z.