mlsanityData to 14 Oct 2026

Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning

arXiv 2111.03189 ↗First posted 4 Nov 2021cs.LG cs.AI cs.ROcited by 26

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning can train policies that effectively perform complex tasks. However for long-horizon tasks, the performance of these methods degrades with horizon, often necessitating reasoning over and chaining lower-level skills. Hierarchical reinforcement learning aims to enable this by providing a bank of low-level skills as action abstractions. Hierarchies can further improve on this by abstracting the space states as well. We posit that a suitable state abstraction should depend on the capabilities of the available lower-level policies. We propose Value Function Spaces: a simple approach that produces such a representation by using the value functions corresponding to each lower-level skill. These value functions capture the affordances of the scene, thus forming a representation that compactly abstracts task relevant information and robustly ignores distractors. Empirical evaluations for maze-solving and robotic manipulation tasks demonstrate that our approach improves long-horizon performance and enables better zero-shot generalization than alternative model-free and model-based methods.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 60 of its 100 references that are in the map. First posted on arXiv 4 Nov 2021.

Builds on

60 of 100 references.

References · 60

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  4. 2015Continuous control with deep reinforcement learningcited by 3,407
  5. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,849
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  7. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  8. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  9. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  10. 2017Hindsight Experience Replaycited by 1,136
  11. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  12. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
38 more
  1. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  2. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  3. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  4. 2016The Option-Critic Architecturecited by 448
  5. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  6. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 404
  7. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 313
  8. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 286
  9. 2016Deep Visual Foresight for Planning Robot Motioncited by 285
  10. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 274
  11. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 274
  12. 2018Visual Reinforcement Learning with Imagined Goalscited by 251
  13. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 231
  14. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 229
  15. 2018Semi-parametric Topological Memory for Navigationcited by 209
  16. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 194
  17. 2019Soft Actor-Critic for Discrete Action Settingscited by 174
  18. 2017DARLA: Improving Zero-Shot Transfer in Reinforcement Learningcited by 149
  19. 2017Self-Supervised Visual Planning with Temporal Skip Connectionscited by 139
  20. 2019Scalable methods for computing state similarity in deterministic Markov Decision Processescited by 133
  21. 2015Model Predictive Path Integral Control using Covariance Variable Importance Samplingcited by 124
  22. 2017Stochastic Neural Networks for Hierarchical Reinforcement Learningcited by 124
  23. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  24. 2017Value Prediction Networkcited by 117
  25. 2018Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Controlcited by 113
  26. 2018Unsupervised Control Through Non-Parametric Discriminative Rewardscited by 102
  27. 2016A Deep Hierarchical Approach to Lifelong Learning in Minecraftcited by 90
  28. 2017Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learningcited by 85
  29. 2020Learning Dexterous Grasping with Object-Centric Visual Affordancescited by 85
  30. 2019InfoBot: Transfer and Exploration via the Information Bottleneckcited by 78
  31. 2016Loss is its own Reward: Self-Supervision for Reinforcement Learningcited by 72
  32. 2018Composable Deep Reinforcement Learning for Robotic Manipulationcited by 72
  33. 2019Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?cited by 68
  34. 2021The Option Keyboard: Combining Skills in Reinforcement Learningcited by 67
  35. 2019Shaping Belief States with Generative Environment Models for RLcited by 66
  36. 2021Hierarchical Reinforcement Learning By Discovering Intrinsic Optionscited by 58
  37. 2018Learning Actionable Representations with Goal-Conditioned Policiescited by 54
  38. 2017Multi-Level Discovery of Deep Optionscited by 51

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 8; 2023: 6; 2024: 10; 2025: 1; 2026: 1.

26 papers.

Sort:

By area:

  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  2. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  3. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 611
  4. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 104
  5. 2023Generative Skill Chaining: Long-Horizon Skill Planning with Diffusion Modelscited by 91
  6. 2022Latent Plans for Task-Agnostic Offline Reinforcement Learningcited by 81
  7. 2023Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agentscited by 67
  8. 2024LoTa-Bench: Benchmarking Language-oriented Task Planners for Embodied Agentscited by 51
  9. 2022Skill-based Model-based Reinforcement Learningcited by 50
  10. 2023LangNav: Language as a Perceptual Representation for Navigationcited by 47
  11. 2024CurricuLLM: Automatic Task Curricula Design for Learning Complex Robot Skills using Large Language Modelscited by 22
  12. 2024RoboMatrix: A Skill-centric Hierarchical Framework for Scalable Robot Task Planning and Execution in Open-Worldcited by 22
14 more
  1. 2022Generating Executable Action Plans with Environmentally-Aware Language Modelscited by 21
  2. 2024Stitching Sub-Trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RLcited by 21
  3. 2022STAP: Sequencing Task-Agnostic Policiescited by 18
  4. 2022GALOIS: Boosting Deep Reinforcement Learning via Generalizable Logic Synthesiscited by 16
  5. 2023Consciousness-Inspired Spatio-Temporal Abstractions for Better Generalization in Reinforcement Learningcited by 4
  6. 2025Compose by Focus: Scene Graph-based Atomic Skillscited by 4
  7. 2024RoboMP2: A Robotic Multimodal Perception-Planning Framework with Multimodal Large Language Modelscited by 3
  8. 2024Learning Planning Abstractions from Languagecited by 3
  9. 2024Exploiting Hybrid Policy in Reinforcement Learning for Interpretable Temporal Logic Manipulationcited by 3
  10. 2023Hierarchical Reinforcement Learning in Complex 3D Environmentscited by 0
  11. 2024Future Predictive Success-or-Failure Classification for Long-Horizon Robotic Taskscited by 0
  12. 2024Goal Exploration via Adaptive Skill Distribution for Goal-Conditioned Reinforcement Learningcited by 0
  13. 2024Learning telic-controllable state representationscited by 0
  14. 2026Training Generalist Navigation Policies via Action Self-Distillationcited by 0