mlsanityData to 7 Oct 2026

A Geometric Perspective on Optimal Representations for Reinforcement Learning

arXiv 1901.11530 ↗First posted 31 Jan 2019cs.LG cs.AI stat.MLcited by 54

Not on the map. Why?

Abstract (arXiv)

We propose a new perspective on representation learning in reinforcement learning based on geometric properties of the space of value functions. We leverage this perspective to provide formal evidence regarding the usefulness of value functions as auxiliary tasks. Our formulation considers adapting the representation to minimize the (linear) approximation of the value function of all stationary policies for a given environment. We show that this optimization reduces to making accurate predictions regarding a special class of value functions which we call adversarial value functions (AVFs). We demonstrate that using value functions as auxiliary tasks corresponds to an expected-error relaxation of our formulation, with AVFs a natural candidate, and identify a close relationship with proto-value functions (Mahadevan, 2005). We highlight characteristics of AVFs and their usefulness as auxiliary tasks in a series of experiments on the four-room domain.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 Jan 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 0; 2021: 10; 2022: 15; 2023: 14; 2024: 9; 2025: 3; 2026: 2.

54 papers.

Sort:

By area:

  1. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  2. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 124
  3. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 116
  4. 2021Model-free Representation Learning and Exploration in Low-rank MDPscited by 82
  5. 2022Denoised MDPs: Learning World Models Better Than the World Itselfcited by 66
  6. 2021On The Effect of Auxiliary Tasks on Representation Dynamicscited by 56
  7. 2023TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learningcited by 56
  8. 2022Efficient Reinforcement Learning in Block MDPs: A Model-free Representation Learning Approachcited by 51
  9. 2021CompilerGym: Robust, Performant Compiler Optimization Environments for AI Researchcited by 41
  10. 2023Proto-Value Networks: Scaling Representation Learning with Auxiliary Taskscited by 41
  11. 2021Learning General Optimal Policies with Graph Neural Networks: Expressive Power, Transparency, and Limitscited by 38
  12. 2019An Optimistic Perspective on Offline Reinforcement Learningcited by 37
38 more
  1. 2021Return-Based Contrastive Representation Learning for Reinforcement Learningcited by 36
  2. 2021The Information Geometry of Unsupervised Reinforcement Learningcited by 36
  3. 2021Which Mutual-Information Representation Learning Objectives are Sufficient for Control?cited by 28
  4. 2022On the Generalization of Representations in Reinforcement Learningcited by 28
  5. 2021Metrics and continuity in reinforcement learningcited by 26
  6. 2022Contrastive UCB: Provably Efficient Contrastive Self-Supervised Learning in Online Reinforcement Learningcited by 26
  7. 2022Investigating the Properties of Neural Network Representations in Reinforcement Learningcited by 24
  8. 2022A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learningcited by 23
  9. 2021Provably Efficient Policy Optimization for Two-Player Zero-Sum Markov Gamescited by 18
  10. 2022Transfer RL across Observation Feature Spaces via Model-Based Regularizationcited by 18
  11. 2024Proto Successor Measure: Representing the Behavior Space of an RL Agentcited by 17
  12. 2024When does Self-Prediction help? Understanding Auxiliary Tasks in Reinforcement Learningcited by 13
  13. 2023Bootstrapped Representations in Reinforcement Learningcited by 11
  14. 2022Exploring through Random Curiosity with General Value Functionscited by 10
  15. 2023Harnessing Discrete Representations For Continual Reinforcement Learningcited by 10
  16. 2022Frustratingly Easy Regularization on Representation Can Boost Deep Reinforcement Learningcited by 8
  17. 2022A Novel Stochastic Gradient Descent Algorithm for Learning Principal Subspacescited by 7
  18. 2021Learning State Representations from Random Deep Action-conditional Predictionscited by 6
  19. 2022The Geometry of Robust Value Functionscited by 6
  20. 2025Physics-informed Temporal Difference Metric Learning for Robot Motion Planningcited by 6
  21. 2023The RL Perceptron: Generalisation Dynamics of Policy Learning in High Dimensionscited by 5
  22. 2022On learning history based policies for controlling Markov decision processescited by 4
  23. 2023PcLast: Discovering Plannable Continuous Latent Statescited by 4
  24. 2024Investigating Pre-Training Objectives for Generalization in Vision-Based Reinforcement Learningcited by 4
  25. 2022Geometric Policy Iteration for Markov Decision Processescited by 3
  26. 2023Towards a Better Understanding of Representation Dynamics under TD-learningcited by 3
  27. 2023Investigating Navigation Strategies in the Morris Water Maze through Deep Reinforcement Learningcited by 3
  28. 2023A Kernel Perspective on Behavioural Metrics for Markov Decision Processescited by 3
  29. 2023Representations and Exploration for Deep Reinforcement Learning using Singular Value Decompositioncited by 2
  30. 2024A Role of Environmental Complexity on Representation Learning in Deep Reinforcement Learning Agentscited by 2
  31. 2024MDP Geometry, Normalization and Reward Balancing Solverscited by 2
  32. 2022Neural Distillation as a State Representation Bottleneck in Reinforcement Learningcited by 1
  33. 2023VIBR: Learning View-Invariant Value Functions for Robust Visual Controlcited by 1
  34. 2023Online Network Source Optimization with Graph-Kernel MABcited by 1
  35. 2024Graph Neural Thompson Samplingcited by 1
  36. 2025Stackelberg Coupling of Online Representation Learning and Reinforcement Learningcited by 1
  37. 2026Task-Induced Representational Invariances Depend on Learning Objective in Deep RLcited by 1
  38. 2023Loss Dynamics of Temporal Difference Reinforcement Learningcited by 0

50 most cited of 54 All 54 (JSON)