mlsanityData to 7 Oct 2026

Ray Interference: a Source of Plateaus in Deep Reinforcement Learning

arXiv 1904.11455 ↗First posted 25 Apr 2019cs.LG cs.AI stat.MLcited by 40

Not on the map. Why?

Abstract (arXiv)

Rather than proposing a new method, this paper investigates an issue present in existing learning algorithms. We study the learning dynamics of reinforcement learning (RL), specifically a characteristic coupling between learning and data generation that arises because RL agents control their future data distribution. In the presence of function approximation, this coupling can lead to a problematic type of 'ray interference', characterized by learning dynamics that sequentially traverse a number of performance plateaus, effectively constraining the agent to learn one thing at a time even when learning in parallel is better. We establish the conditions under which ray interference occurs, show its relation to saddle points and obtain the exact learning dynamics in a restricted setting. We characterize a number of its properties and discuss possible remedies.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 5; 2023: 6; 2024: 7; 2025: 5; 2026: 9.

40 papers.

Sort:

By area:

  1. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  2. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  3. 2022Multi-Task Learning as a Bargaining Gamecited by 209
  4. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 201
  5. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 124
  6. 2021Continual World: A Robotic Benchmark For Continual Reinforcement Learningcited by 112
  7. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspectivecited by 70
  8. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learningcited by 64
  9. 2026POPE: Learning to Reason on Hard Problems via Privileged On-Policy Explorationcited by 39
  10. 2021Improving Multilingual Translation by Representation and Gradient Regularizationcited by 31
  11. 2026Maximum Likelihood Reinforcement Learningcited by 29
  12. 2023Vanishing Gradients in Reinforcement Finetuning of Language Modelscited by 26
28 more
  1. 2024A Study of Plasticity Loss in On-Policy Deep Reinforcement Learningcited by 26
  2. 2024Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problemcited by 23
  3. 2024A social path to human-like artificial intelligencecited by 20
  4. 2022Improving Policy Optimization with Generalist-Specialist Learningcited by 19
  5. 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Modelscited by 17
  6. 2026Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixescited by 16
  7. 2021Catastrophic Interference in Reinforcement Learning: A Solution Based on Context Division and Knowledge Distillationcited by 15
  8. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 14
  9. 2023Auxiliary Learning as an Asymmetric Bargaining Gamecited by 12
  10. 2021Model-based micro-data reinforcement learning: what are the crucial model properties and which model to choose?cited by 11
  11. 2022Switch Trajectory Transformer with Distributional Value Approximation for Multi-Task Reinforcement Learningcited by 11
  12. 2022Learning Dynamics and Generalization in Reinforcement Learningcited by 11
  13. 2024Bayesian Uncertainty for Gradient Aggregation in Multi-Task Learningcited by 11
  14. 2026Vero: An Open RL Recipe for General Visual Reasoningcited by 10
  15. 2025NoProp: Training Neural Networks without Full Back-propagation or Full Forward-propagationcited by 8
  16. 2023QMP: Q-switch Mixture of Policies for Multi-Task Behavior Sharingcited by 6
  17. 2026Why Pass@k Optimization Can Degrade Pass@1: Prompt Interference in LLM Post-trainingcited by 6
  18. 2023Measuring and Mitigating Interference in Reinforcement Learningcited by 5
  19. 2026IsoCompute Playbook: Optimally Scaling Sampling Compute for LLM RLcited by 5
  20. 2025Reasoner for Real-World Event Detection: Scaling Reinforcement Learning via Adaptive Perplexity-Aware Sampling Strategycited by 3
  21. 2021A Generalised Inverse Reinforcement Learning Frameworkcited by 2
  22. 2023Scaling Goal-based Exploration via Pruning Proto-goalscited by 2
  23. 2024Robust Analysis of Multi-Task Learning Efficiency: New Benchmarks on Light-Weighed Backbones and Effective Measurement of Multi-Task Learning Challenges by Feature Disentanglementcited by 1
  24. 2024Toward Debugging Deep Reinforcement Learning Programs with RLExplorercited by 1
  25. 2026When Errors Can Be Beneficial: A Categorization of Imperfect Rewards for Policy Gradientcited by 1
  26. 2026Revisiting Policy Gradients for Restricted Policy Classes: Escaping Myopic Local Optima with k-step Policy Gradientscited by 1
  27. 2023Projected Task-Specific Layers for Multi-Task Reinforcement Learningcited by 0
  28. 2026TelecomGPT-R1: Unified Post-Training for Reasoning Across Heterogeneous Telecom Taskscited by 0