mlsanityData to 14 Oct 2026

Policy Optimization in a Noisy Neighborhood: On Return Landscapes in Continuous Control

arXiv 2309.14597 ↗First posted 26 Sep 2023cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Deep reinforcement learning agents for continuous control are known to exhibit significant instability in their performance over time. In this work, we provide a fresh perspective on these behaviors by studying the return landscape: the mapping between a policy and a return. We find that popular algorithms traverse noisy neighborhoods of this landscape, in which a single update to the policy parameters leads to a wide range of returns. By taking a distributional view of these returns, we map the landscape, characterizing failure-prone regions of policy space and revealing a hidden dimension of policy quality. We show that the landscape exhibits surprising structure by finding simple paths in parameter space which improve the stability of a policy. To conclude, we develop a distribution-aware procedure which finds such paths, navigating away from noisy neighborhoods in order to improve the robustness of a policy. Taken together, our results provide new insight into the optimization, evaluation, and design of agents.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 24 of its 48 references that are in the map. First posted on arXiv 26 Sep 2023.

Builds on

24 of 48 references.

References · 24

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  6. 2018DeepMind Control Suitecited by 956
  7. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  8. 2017Deep Reinforcement Learning that Matterscited by 754
  9. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 637
  10. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 588
  11. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  12. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 330
12 more
  1. 2014Qualitatively characterizing neural network optimization problemscited by 188
  2. 2018Understanding the impact of entropy on policy optimizationcited by 176
  3. 2016Topology and Geometry of Half-Rectified Network Optimizationcited by 102
  4. 2019PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaoscited by 56
  5. 2019Measuring the Reliability of Reinforcement Learning Algorithmscited by 51
  6. 2021Training Larger Networks for Deep Reinforcement Learningcited by 38
  7. 2022The Phenomenon of Policy Churncited by 28
  8. 2019A Hitchhiker's Guide to Statistical Comparisons of Reinforcement Learning Algorithmscited by 27
  9. 2019Smoothing Policies and Safe Policy Gradientscited by 21
  10. 2022Cliff Diving: Exploring Reward Surfaces in Reinforcement Learning Environmentscited by 9
  11. 2020Visualizing the Loss Landscape of Actor Critic Methods with Applications in Inventory Optimizationcited by 5
  12. 2019Never Worse, Mostly Better: Stable Policy Improvement in Deep Reinforcement Learningcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 3; 2026: 5.

11 papers.

  1. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 102
  2. 2024Do Transformer World Models Give Better Policy Gradients?cited by 10
  3. 2025Relative Entropy Pathwise Policy Optimizationcited by 7
  4. 2024Enhancing Robustness in Deep Reinforcement Learning: A Lyapunov Exponent Approachcited by 4
  5. 2025Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learningcited by 2
  6. 2026SHAPO: Sharpness-Aware Policy Optimization for Safe Explorationcited by 2
  7. 2026Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learningcited by 1
  8. 2025Efficient On-Policy Reinforcement Learning via Exploration of Sparse Parameter Spacecited by 0
  9. 2026Moments Matter:Stabilizing Policy Optimization using Return Distributionscited by 0
  10. 2026Visualizing Critic Match Loss Landscapes for Interpretation of Online Reinforcement Learning Control Algorithmscited by 0
  11. 2026Continual Reinforcement Learning with Neuroevolutioncited by 0