mlsanityData to 14 Oct 2026

Can Agents Run Relay Race with Strangers? Generalization of RL to Out-of-Distribution Trajectories

arXiv 2304.13424 ↗First posted 26 Apr 2023cs.LG cs.AI cs.ROcited by 9

Not on the map. Why?

Abstract (arXiv)

In this paper, we define, evaluate, and improve the ``relay-generalization'' performance of reinforcement learning (RL) agents on the out-of-distribution ``controllable'' states. Ideally, an RL agent that generally masters a task should reach its goal starting from any controllable state of the environment instead of memorizing a small set of trajectories. For example, a self-driving system should be able to take over the control from humans in the middle of driving and must continue to drive the car safely. To practically evaluate this type of generalization, we start the test agent from the middle of other independently well-trained stranger agents' trajectories. With extensive experimental evaluation, we show the prevalence of generalization failure on controllable states from stranger agents. For example, in the Humanoid environment, we observed that a well-trained Proximal Policy Optimization (PPO) agent, with only 3.9% failure rate during regular testing, failed on 81.6% of the states generated by well-trained stranger PPO agents. To improve "relay generalization," we propose a novel method called Self-Trajectory Augmentation (STA), which will reset the environment to the agent's old states according to the Q function during training. After applying STA to the Soft Actor Critic's (SAC) training procedure, we reduced the failure rate of SAC under relay-evaluation by more than three times in most settings without impacting agent performance and increasing the needed number of environment interactions. Our code is available at https://github.com/lan-lc/STA.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 33 of its 42 references that are in the map. First posted on arXiv 26 Apr 2023.

Builds on

33 of 42 references.

References · 33

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  6. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  7. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  8. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
  9. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  10. 2017Robust Adversarial Reinforcement Learningcited by 451
  11. 2017Population Based Training of Neural Networkscited by 367
  12. 2018Quantifying Generalization in Reinforcement Learningcited by 352
21 more
  1. 2017Adversarial Attacks on Neural Network Policiescited by 320
  2. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 275
  3. 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 244
  4. 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 238
  5. 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 212
  6. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 209
  7. 2018Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networkscited by 174
  8. 2021Robust Reinforcement Learning on State Observations with Learned Optimal Adversarycited by 135
  9. 2018Assessing Generalization in Deep Reinforcement Learningcited by 133
  10. 2019Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environmentscited by 119
  11. 2018A Dissection of Overfitting and Generalization in Continuous Reinforcement Learningcited by 87
  12. 2020Challenges and Countermeasures for Adversarial Attacks on Deep Reinforcement Learningcited by 61
  13. 2021The Difficulty of Passive Learning in Deep Reinforcement Learningcited by 50
  14. 2021Robust Stochastic Linear Contextual Bandits Under Adversarial Attackscited by 33
  15. 2018Continual Reinforcement Learning with Complex Synapsescited by 32
  16. 2018Natural Environment Benchmarks for Reinforcement Learningcited by 32
  17. 2021Efficient Local Planning with Linear Function Approximationcited by 23
  18. 2019Investigating Generalisation in Continuous Deep Reinforcement Learningcited by 18
  19. 2022Are AlphaZero-like Agents Robust to Adversarial Perturbations?cited by 14
  20. 2021Adaptable Agent Populations via a Generative Model of Policiescited by 12
  21. 2023Sample Efficient Deep Reinforcement Learning via Local Planningcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 3; 2025: 3; 2026: 1.

9 papers.

  1. 2024FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learningcited by 26
  2. 2023Bad Habits: Policy Confounding and Out-of-Trajectory Generalization in RLcited by 10
  3. 2025Exploring Expert Failures Improves LLM Agent Tuningcited by 9
  4. 2025LaGEA: Language Guided Embodied Agents for Robotic Manipulationcited by 2
  5. 2026MARVL: Multi-Stage Guidance for Robotic Manipulation via Vision-Language Modelscited by 2
  6. 2023Modifying RL Policies with Imagined Actions: How Predictable Policies Can Enable Users to Perform Novel Taskscited by 1
  7. 2024RL in Markov Games with Independent Function Approximation: Improved Sample Complexity Bound under the Local Access Modelcited by 0
  8. 2024Imagining In-distribution States: How Predictable Robot Behavior Can Enable User Control Over Learned Policiescited by 0
  9. 2025Breaking Habits: On the Role of the Advantage Function in Learning Causal State Representationscited by 0