mlsanityData to 7 Oct 2026

Generalization in Visual Reinforcement Learning with the Reward Sequence Distribution

arXiv 2302.09601 ↗First posted 19 Feb 2023cs.LG cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

Generalization in partially observed markov decision processes (POMDPs) is critical for successful applications of visual reinforcement learning (VRL) in real scenarios. A widely used idea is to learn task-relevant representations that encode task-relevant information of common features in POMDPs, i.e., rewards and transition dynamics. As transition dynamics in the latent state space -- which are task-relevant and invariant to visual distractions -- are unknown to the agents, existing methods alternatively use transition dynamics in the observation space to extract task-relevant information in transition dynamics. However, such transition dynamics in the observation space involve task-irrelevant visual distractions, degrading the generalization performance of VRL methods. To tackle this problem, we propose the reward sequence distribution conditioned on the starting observation and the predefined subsequent action sequence (RSD-OA). The appealing features of RSD-OA include that: (1) RSD-OA is invariant to visual distractions, as it is conditioned on the predefined subsequent action sequence without task-irrelevant information from transition dynamics, and (2) the reward sequence captures long-term task-relevant information in both rewards and transition dynamics. Experiments demonstrate that our representation learning approach based on RSD-OA significantly improves the generalization performance on unseen environments, outperforming several state-of-the-arts on DeepMind Control tasks with visual distractions.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 38 of its 46 references that are in the map. First posted on arXiv 19 Feb 2023.

Builds on

38 of 46 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,141
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  7. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  8. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  9. 2018DeepMind Control Suitecited by 935
  10. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  11. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  12. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
26 more
  1. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  2. 2020Reinforcement Learning with Augmented Datacited by 452
  3. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  4. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
  5. 2019Provably efficient RL with Rich Observations via Latent State Decodingcited by 169
  6. 2022TransDreamer: Reinforcement Learning with Transformer World Modelscited by 143
  7. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  8. 2020Self-Supervised Policy Adaptation during Deploymentcited by 121
  9. 2018Generalization and Regularization in DQNcited by 117
  10. 2020Invariant Causal Prediction for Block MDPscited by 104
  11. 2019Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learningcited by 102
  12. 2019Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneckcited by 101
  13. 2021The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixelscited by 97
  14. 2019Observational Overfitting in Reinforcement Learningcited by 90
  15. 2021Decoupling Value and Policy for Generalization in Reinforcement Learningcited by 81
  16. 2020Deep Reinforcement and InfoMax Learningcited by 79
  17. 2020Automatic Data Augmentation for Generalization in Deep Reinforcement Learningcited by 71
  18. 2021SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policiescited by 42
  19. 2020Invariant Policy Optimization: Towards Stronger Generalization in Reinforcement Learningcited by 37
  20. 2022Attention Consistency on Visual Corruptions for Single-Source Domain Generalizationcited by 37
  21. 2021DRIBO: Robust Deep Reinforcement Learning via Multi-View Information Bottleneckcited by 31
  22. 2019A Characteristic Function Approach to Deep Implicit Generative Modelingcited by 25
  23. 2021Sample-Efficient Reinforcement Learning via Conservative Model-Based Actor-Criticcited by 23
  24. 2021Cross-Trajectory Representation Learning for Zero-Shot Generalization in RLcited by 20
  25. 2022Learning Task-relevant Representations for Generalization via Characteristic Functions of Reward Sequence Distributionscited by 15
  26. 2021Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddingscited by 12

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1; 2025: 2.

4 papers.

  1. 2025Apollo-MILP: An Alternating Prediction-Correction Neural Solving Framework for Mixed-Integer Linear Programmingcited by 16
  2. 2024Scalable and Effective Arithmetic Tree Generation for Adder and Multiplier Designscited by 9
  3. 2023State Sequences Prediction via Fourier Transform for Representation Learningcited by 6
  4. 2025DRARL: Disengagement-Reason-Augmented Reinforcement Learning for Efficient Improvement of Autonomous Driving Policycited by 1