mlsanityData to 7 Oct 2026

Contextualize Me -- The Case for Context in Reinforcement Learning

arXiv 2202.04500 ↗First posted 9 Feb 2022cs.LGcited by 41

Not on the map. Why?

Abstract (arXiv)

While Reinforcement Learning ( RL) has made great strides towards solving increasingly complicated problems, many algorithms are still brittle to even slight environmental changes. Contextual Reinforcement Learning (cRL) provides a framework to model such changes in a principled manner, thereby enabling flexible, precise and interpretable task specification and generation. Our goal is to show how the framework of cRL contributes to improving zero-shot generalization in RL through meaningful benchmarks and structured reasoning about generalization tasks. We confirm the insight that optimal behavior in cRL requires context information, as in other related areas of partial observability. To empirically validate this in the cRL framework, we provide various context-extended versions of common RL environments. They are part of the first benchmark library, CARL, designed for generalization based on cRL extensions of popular benchmarks, which we propose as a testbed to further study general agents. We show that in the contextual setting, even simple RL environments become challenging - and that naive solutions are not enough to generalize across complex context spaces.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 67 of its 104 references that are in the map. First posted on arXiv 9 Feb 2022.

Builds on

67 of 104 references.

References · 67

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  6. 2016OpenAI Gymcited by 1,684
  7. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  8. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  9. 2018DeepMind Control Suitecited by 935
  10. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  11. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  12. 2017Deep Reinforcement Learning that Matterscited by 727
38 more
  1. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  2. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 494
  3. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  4. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  5. 2017Robust Adversarial Reinforcement Learningcited by 439
  6. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  7. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  8. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  9. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
  10. 2016Learning to reinforcement learncited by 369
  11. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  12. 2017Population Based Training of Neural Networkscited by 351
  13. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  14. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  15. 2020Transfer Learning in Deep Reinforcement Learning: A Surveycited by 259
  16. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
  17. 2015Contextual Markov Decision Processescited by 213
  18. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
  19. 2017Teacher-Student Curriculum Learningcited by 197
  20. 2020The NetHack Learning Environmentcited by 196
  21. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 188
  22. 2021Reinforcement Learning with Prototypical Representationscited by 179
  23. 2019Provably efficient RL with Rich Observations via Latent State Decodingcited by 169
  24. 2021Multi-Task Reinforcement Learning with Context-based Representationscited by 169
  25. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
  26. 2019Active Domain Randomizationcited by 161
  27. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 145
  28. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 138
  29. 2021Robust Reinforcement Learning on State Observations with Learned Optimal Adversarycited by 133
  30. 2021Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observabilitycited by 102
  31. 2021MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Researchcited by 96
  32. 2020Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learningcited by 95
  33. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
  34. 2013Hidden Parameter Markov Decision Processes: A Semiparametric Regression Approach for Discovering Latent Task Parametrizationscited by 68
  35. 2017Hyperparameter Importance Across Datasetscited by 62
  36. 2020Provably Efficient Online Hyperparameter Optimization with Population-Based Banditscited by 61
  37. 2021On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learningcited by 59
  38. 2021Learning Task Informed Abstractionscited by 58

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 7; 2024: 14; 2025: 9; 2026: 9.

41 papers.

Sort:

By area:

  1. 2024Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Taskscited by 40
  2. 2024ODRL: A Benchmark for Off-Dynamics Reinforcement Learningcited by 25
  3. 2023Universal Morphology Control via Contextual Modulationcited by 21
  4. 2023Dynamics Generalisation in Reinforcement Learning via Adaptive Context-Aware Policiescited by 18
  5. 2024Zero-Shot Reinforcement Learning via Function Encoderscited by 17
  6. 2024Dreaming of Many Worlds: Learning Contextual World Models Aids Zero-Shot Generalizationcited by 16
  7. 2024AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformerscited by 14
  8. 2023Transformers in Reinforcement Learning: A Surveycited by 11
  9. 2022The Impact of Task Underspecification in Evaluating Deep Reinforcement Learningcited by 10
  10. 2024Model-Based Transfer Learning for Contextual Reinforcement Learningcited by 7
  11. 2023Temporal Transfer Learning for Traffic Optimization with Coarse-grained Advisory Autonomycited by 6
  12. 2024Hierarchical Transformers are Efficient Meta-Reinforcement Learnerscited by 6
29 more
  1. 2024IntersectionZoo: Eco-driving for Benchmarking Multi-Agent Contextual Reinforcement Learningcited by 6
  2. 2024Generalizing Cooperative Eco-driving via Multi-residual Task Learningcited by 5
  3. 2025Dynamics-Aligned Latent Imagination in Contextual World Models for Zero-Shot Generalizationcited by 5
  4. 2023Deep Reinforcement Learning with Explicit Context Representationcited by 3
  5. 2024ARLBench: Flexible and Efficient Benchmarking for Hyperparameter Optimization in Reinforcement Learningcited by 3
  6. 2024Inferring Behavior-Specific Context Improves Zero-Shot Generalization in Reinforcement Learningcited by 2
  7. 2024GRAM: Generalization in Deep RL with a Robust Adaptation Modulecited by 2
  8. 2025Pessimism Principle Can Be Effective: Towards a Framework for Zero-Shot Transfer Reinforcement Learningcited by 2
  9. 2025Adaptive Shielding for Safe Reinforcement Learning under Hidden-Parameter Dynamics Shiftscited by 2
  10. 2026Structure Detection for Contextual Reinforcement Learningcited by 2
  11. 2022Tri-Attention: Explicit Context-Aware Attention Mechanism for Natural Language Processingcited by 1
  12. 2023Contextual Pre-planning on Reward Machine Abstractions for Enhanced Transfer in Deep Reinforcement Learningcited by 1
  13. 2023Dyadic Reinforcement Learningcited by 1
  14. 2025Multi-residual Mixture of Experts Learning for Cooperative Control in Multi-vehicle Systemscited by 1
  15. 2025CG-TTRL: Context-Guided Test-Time Reinforcement Learning for On-Device Large Language Modelscited by 1
  16. 2026Contextual Intelligence The Next Leap for Reinforcement Learningcited by 1
  17. 2026Contextual Multi-Task Reinforcement Learning for Autonomous Reef Monitoringcited by 1
  18. 2024Adaptive World Models: Learning Behaviors by Latent Imagination Under Non-Stationaritycited by 0
  19. 2024A Research Agenda for Usability and Generalisation in Reinforcement Learningcited by 0
  20. 2025Multi-Objective Planning with Contextual Lexicographic Reward Preferencescited by 0
  21. 2025Zero-Shot Context Generalization in Reinforcement Learning from Few Training Contextscited by 0
  22. 2025Observations Meet Actions: Learning Control-Sufficient Representations for Robust Policy Generalizationcited by 0
  23. 2025Context-Aware Model-Based Reinforcement Learning for Autonomous Racingcited by 0
  24. 2026Self Paced Gaussian Contextual Reinforcement Learningcited by 0
  25. 2026Latent Q-Barrier Shielding for Safe In-Context Reinforcement Learningcited by 0
  26. 2026Post-Hoc Robustness for Model-Based Reinforcement Learningcited by 0
  27. 2026Formalizing Task-Space Complexity for Zero-Shot Generalizationcited by 0
  28. 2026Task Specialization Fine-Tuning for Contextual Reinforcement Learningcited by 0
  29. 2026Dynamic Context Scheduling: Learning Beyond the Static Universecited by 0