mlsanityData to 7 Oct 2026

Multi-Horizon Representations with Hierarchical Forward Models for Reinforcement Learning

arXiv 2206.11396 ↗First posted 22 Jun 2022cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Learning control from pixels is difficult for reinforcement learning (RL) agents because representation learning and policy learning are intertwined. Previous approaches remedy this issue with auxiliary representation learning tasks, but they either do not consider the temporal aspect of the problem or only consider single-step transitions, which may cause learning inefficiencies if important environmental changes take many steps to manifest. We propose Hierarchical k-Step Latent (HKSL), an auxiliary task that learns multiple representations via a hierarchy of forward models that learn to communicate and an ensemble of n-step critics that all operate at varying magnitudes of step skipping. We evaluate HKSL in a suite of 30 robotic control tasks with and without distractors and a task of our creation. We find that HKSL either converges to higher or optimal episodic returns more quickly than several alternative representation learning approaches. Furthermore, we find that HKSL's representations capture task-relevant details accurately across timescales (even in the presence of distractors) and that communication channels between hierarchy levels organize information based on both sides of the communication process, both of which improve sample efficiency.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 55 of its 69 references that are in the map. First posted on arXiv 22 Jun 2022.

Builds on

55 of 69 references.

References · 55

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  4. 2016Layer Normalizationcited by 5,579
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  8. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  9. 2018World Modelscited by 1,444
  10. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  11. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  12. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
38 more
  1. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  2. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  3. 2020Mastering Atari with Discrete World Modelscited by 990
  4. 2018DeepMind Control Suitecited by 935
  5. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  6. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  7. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  8. 2017Deep Reinforcement Learning that Matterscited by 727
  9. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  10. 2020Reinforcement Learning with Augmented Datacited by 452
  11. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  12. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  13. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  14. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  15. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  16. 2018Large-Scale Study of Curiosity-Driven Learningcited by 277
  17. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  18. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  19. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
  20. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
  21. 2019Unsupervised State Representation Learning in Ataricited by 144
  22. 2020RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environmentscited by 135
  23. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  24. 2019Improved Conditional VRNNs for Video Predictioncited by 104
  25. 2016Hierarchical Multiscale Recurrent Neural Networkscited by 102
  26. 2021The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixelscited by 97
  27. 2017Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Controlcited by 92
  28. 2020The Conditional Entropy Bottleneckcited by 75
  29. 2019VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generationcited by 74
  30. 2014A Clockwork RNNcited by 70
  31. 2017HARP: Hierarchical Representation Learning for Networkscited by 58
  32. 2020Predictive Information Accelerates Learning in RLcited by 56
  33. 2019Discovery of Useful Questions as Auxiliary Taskscited by 45
  34. 2019Filter Response Normalization Layer: Eliminating Batch Dependence in the Training of Deep Neural Networkscited by 45
  35. 2021AW-Opt: Learning Robotic Skills with Imitation and Reinforcement at Scalecited by 43
  36. 2021Clockwork Variational Autoencoderscited by 40
  37. 2019Variational Temporal Abstractioncited by 37
  38. 2019CHiVE: Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Networkcited by 25

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 4.

7 papers.

  1. 2025A Survey of State Representation Learning for Deep Reinforcement Learningcited by 17
  2. 2024Multi-view Disentanglement for Reinforcement Learning with Multiple Camerascited by 8
  3. 2025Studying the Interplay Between the Actor and Critic Representations in Reinforcement Learningcited by 7
  4. 2025Enhancing Tactile-based Reinforcement Learning for Robotic Controlcited by 5
  5. 2025PvP: Data-Efficient Humanoid Robot Learning with Proprioceptive-Privileged Contrastive Representationscited by 2
  6. 2024Highway Graph to Accelerate Reinforcement Learningcited by 0
  7. 2024Skill-aware Mutual Information Optimisation for Generalisation in Reinforcement Learningcited by 0