mlsanityData to 7 Oct 2026

Rollout Total Correlation for Deep Reinforcement Learning

arXiv 2209.05333 ↗First posted 12 Sep 2022cs.LG cs.ROcited by 0

Not on the map. Why?

Abstract (arXiv)

Learning task-relevant representations is crucial for reinforcement learning. Recent approaches aim to learn such representations by improving the temporal consistency in the observed transitions. However, they only consider individual transitions and can fail to achieve long-term consistency. Instead, we argue that capturing aspects of the state that correlate with other states and actions of the trajectory---even more distant in the future---could further help in extracting task-relevant information. Hence, in this paper we investigate how to learn representations by maximizing the rollout total correlation, the correlation among all learned representations and actions within the trajectories produced by the agent. For improving rollout total correlation, we propose to combine two complementary lower bounds based on a generative and a discriminative model, combined with a simple and effective technique of chunk-wise mini-batching. Furthermore, we propose an intrinsic reward based on the learned representation for better exploration. Experimental evaluations on a set of challenging image-based simulated control tasks show that our method achieves better sample efficiency, and robustness to both white noise and natural video backgrounds compared to leading baselines.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 26 of its 43 references that are in the map. First posted on arXiv 12 Sep 2022.

Builds on

26 of 43 references.

References · 26

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  2. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2017The Kinetics Human Action Video Datasetcited by 1,894
  5. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  6. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  7. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  8. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  9. 2018DeepMind Control Suitecited by 935
  10. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  11. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  12. 2020Planning to Explore via Self-Supervised World Modelscited by 365
14 more
  1. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  2. 2021Reinforcement Learning with Prototypical Representationscited by 179
  3. 2019Scalable methods for computing state similarity in deterministic Markov Decision Processescited by 133
  4. 2022BYOL-Explore: Exploration by Bootstrapped Predictioncited by 71
  5. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 57
  6. 2020Predictive Information Accelerates Learning in RLcited by 56
  7. 2020Masked Contrastive Representation Learning for Reinforcement Learningcited by 32
  8. 2021DRIBO: Robust Deep Reinforcement Learning via Multi-View Information Bottleneckcited by 31
  9. 2021Robust Predictable Controlcited by 29
  10. 2020Novelty Search in Representational Space for Sample Efficient Explorationcited by 26
  11. 2021Dynamic Bottleneck for Robust Self-Supervised Explorationcited by 21
  12. 2022RoMFAC: A robust mean-field actor-critic reinforcement learning against adversarial perturbations on statescited by 16
  13. 2022Integrating Contrastive Learning with Dynamic Models for Reinforcement Learning from Imagescited by 12
  14. 2022Approximate Policy Iteration with Bisimulation Metricscited by 11

Built on by

Not cited yet.