mlsanityData to 7 Oct 2026

Disentangling Recognition and Decision Regrets in Image-Based Reinforcement Learning

arXiv 2409.13108 ↗First posted 19 Sep 2024cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In image-based reinforcement learning (RL), policies usually operate in two steps: first extracting lower-dimensional features from raw images (the "recognition" step), and then taking actions based on the extracted features (the "decision" step). Extracting features that are spuriously correlated with performance or irrelevant for decision-making can lead to poor generalization performance, known as observational overfitting in image-based RL. In such cases, it can be hard to quantify how much of the error can be attributed to poor feature extraction vs. poor decision-making. To disentangle the two sources of error, we introduce the notions of recognition regret and decision regret. Using these notions, we characterize and disambiguate the two distinct causes behind observational overfitting: over-specific representations, which include features that are not needed for optimal decision-making (leading to high decision regret), vs. under-specific representations, which only include a limited set of features that were spuriously correlated with performance during training (leading to high recognition regret). Finally, we provide illustrative examples of observational overfitting due to both over-specific and under-specific representations in maze environments and the Atari game Pong.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 24 of its 30 references that are in the map. First posted on arXiv 19 Sep 2024.

Builds on

24 of 30 references.

References · 24

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  3. 2017Robust Adversarial Reinforcement Learningcited by 439
  4. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  5. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 320
  6. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  7. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  8. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  9. 2018A Study on Overfitting in Deep Reinforcement Learningcited by 170
  10. 2018Assessing Generalization in Deep Reinforcement Learningcited by 132
  11. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  12. 2019Observational Overfitting in Reinforcement Learningcited by 90
12 more
  1. 2018A Dissection of Overfitting and Generalization in Continuous Reinforcement Learningcited by 86
  2. 2021Decoupling Value and Policy for Generalization in Reinforcement Learningcited by 81
  3. 2017Towards Generalization and Simplicity in Continuous Controlcited by 65
  4. 2020The Value-Improvement Path: Towards Better Representations for Reinforcement Learningcited by 60
  5. 2018Gotta Learn Fast: A New Benchmark for Generalization in RLcited by 59
  6. 2019A Geometric Perspective on Optimal Representations for Reinforcement Learningcited by 54
  7. 2022Efficient Reinforcement Learning in Block MDPs: A Model-free Representation Learning Approachcited by 51
  8. 2020Invariant Policy Optimization: Towards Stronger Generalization in Reinforcement Learningcited by 37
  9. 2022skrl: Modular and Flexible Library for Reinforcement Learningcited by 35
  10. 2020Representations for Stable Off-Policy Reinforcement Learningcited by 31
  11. 2022On the Generalization of Representations in Reinforcement Learningcited by 28
  12. 2022Investigating the Properties of Neural Network Representations in Reinforcement Learningcited by 24

Built on by

Not cited yet.