Learning in Low-Dimensional Subspaces: Orthogonal Bottlenecks for Reinforcement Learning

arXiv 2605.26012 ↗First posted 25 May 2026cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Deep reinforcement learning (RL) agents commonly rely on high-dimensional neural representations, despite growing evidence that task-relevant value and policy structure may be intrinsically low-dimensional. In this work, we present a simple yet effective representation-level prior that inserts a fixed orthonormal projection to constrain encoder features to a low-dimensional subspace, requiring no auxiliary objectives, pretraining, or changes to the underlying RL algorithm. Under a linear realizability assumption, we prove that when the bottleneck dimension exceeds the intrinsic rank of the optimal value function in feature space, the bottleneck preserves expressivity and leaves the induced gradient dynamics unchanged up to an equivalent low-dimensional parameterization. Empirically, we find that across both single and multi-task benchmarks, baseline performance is either matched or improved once the bottleneck dimension exceeds a small task-dependent threshold; in many cases, value representations can be compressed to extremely low dimensions without loss, and the minimal sufficient dimension depends far more on environment complexity than encoder width. In addition, we analyze representation geometry and find that orthogonal bottlenecks stabilize feature norms and are associated with higher effective rank. Together, these results support a representation-space interpretation of the manifold hypothesis in reinforcement learning and position orthogonal bottlenecks as a lightweight, architecture-agnostic mechanism for shaping RL representations.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 49 of its 66 references that are in the map. First posted on arXiv 25 May 2026.

Builds on

49 of 66 references.

Closest ancestors · 7

Same direction, newest first

Other references · 42

2017Proximal Policy Optimization Algorithmscited by 15,215
2020Gradient Surgery for Multi-Task Learningcited by 1,256
2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
2020Object-Centric Learning with Slot Attentioncited by 876
2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
2019Multi-Object Representation Learning with Iterative Variational Inferencecited by 360
2015Unitary Evolution Recurrent Neural Networkscited by 294
2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176

30 more

2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 174
2023Understanding plasticity in neural networkscited by 174
2020Multi-Task Reinforcement Learning with Soft Modularizationcited by 143
2019Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning?cited by 140
2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
2017Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practicecited by 132
2019Learning with Good Feature Representations in Bandits and in RL with a Generative Modelcited by 127
2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 125
2018Variational Option Discovery Algorithmscited by 122
2022Discovered Policy Optimisationcited by 104
2020Continual Learning in Low-rank Orthogonal Subspacescited by 99
2017Meta Learning Shared Hierarchiescited by 93
2018The Emergence of Spectral Universality in Deep Networkscited by 74
2020Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networkscited by 73
2019Orthogonal Deep Neural Networkscited by 72
2023Manifold learning: what, how, and whycited by 68
2022The State of Sparse Training in Deep Reinforcement Learningcited by 61
2019A Mean Field Theory of Batch Normalizationcited by 58
2023Multi-Task Reinforcement Learning with Mixture of Orthogonal Expertscited by 49
2018Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networkscited by 45
2022Atari-5: Distilling the Arcade Learning Environment down to Five Gamescited by 44
2016Recurrent Orthogonal Networks and Long-Memory Taskscited by 38
2020Controllable Orthogonalization in Training DNNscited by 30
2024Entity-Centric Reinforcement Learning for Object Manipulation from Pixelscited by 27
2022Residual Skill Policies: Learning an Adaptable Skill-based Action Space for Reinforcement Learning for Roboticscited by 24
2021Factor Analysis, Probabilistic Principal Component Analysis, Variational Inference, and Variational Autoencoder: Tutorial and Surveycited by 21
2021Policy Manifold Search: Exploring the Manifold Hypothesis for Diversity-based Neuroevolutioncited by 20
2019Dynamical Isometry and a Mean Field Theory of LSTMs and GRUscited by 17
2018VPE: Variational Policy Embedding for Transfer Reinforcement Learningcited by 8
2022Reward-Free Policy Space Compression for Reinforcement Learningcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way