Analyzing Generalization in Policy Networks: A Case Study with the Double-Integrator System
Not on the map. Why?
Abstract (arXiv)
Extensive utilization of deep reinforcement learning (DRL) policy networks in diverse continuous control tasks has raised questions regarding performance degradation in expansive state spaces where the input state norm is larger than that in the training environment. This paper aims to uncover the underlying factors contributing to such performance deterioration when dealing with expanded state spaces, using a novel analysis technique known as state division. In contrast to prior approaches that employ state division merely as a post-hoc explanatory tool, our methodology delves into the intrinsic characteristics of DRL policy networks. Specifically, we demonstrate that the expansion of state space induces the activation function tanh to exhibit saturability, resulting in the transformation of the state division boundary from nonlinear to linear. Our analysis centers on the paradigm of the double-integrator system, revealing that this gradual shift towards linearity imparts a control behavior reminiscent of bang-bang control. However, the inherent linearity of the division boundary prevents the attainment of an ideal bang-bang control, thereby introducing unavoidable overshooting. Our experimental investigations, employing diverse RL algorithms, establish that this performance phenomenon stems from inherent attributes of the DRL policy network, remaining consistent across various optimization algorithms.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 27 of its 50 references that are in the map. First posted on arXiv 16 Dec 2023.
Builds on
27 of 50 references.
References · 27
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
- 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
- 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
- 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
- 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
- 2020Reinforcement Learning with Augmented Datacited by 468
- 2018Generalizing to Unseen Domains via Adversarial Data Augmentationcited by 451
- 2018Quantifying Generalization in Reinforcement Learningcited by 352
- 2017Deep Reinforcement Learning: An Overviewcited by 333
- 2018SpotTune: Transfer Learning through Adaptive Fine-tuningcited by 168
- 2019Complexity of Linear Regions in Deep Networkscited by 167
15 more
- 2021FSDR: Frequency Space Domain Randomization for Domain Generalizationcited by 164
- 2017Bounding and Counting Linear Regions of Deep Neural Networkscited by 161
- 2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 138
- 2018Assessing Generalization in Deep Reinforcement Learningcited by 133
- 2020Self-Supervised Policy Adaptation during Deploymentcited by 123
- 2013On the number of response regions of deep feed forward networks with piece-wise linear activationscited by 94
- 2020Improving Generalization in Reinforcement Learning with Mixture Regularizationcited by 81
- 2020Automatic Data Augmentation for Generalization in Deep Reinforcement Learningcited by 71
- 2019Improving Generalization in Meta Reinforcement Learning using Learned Objectivescited by 68
- 2019Robust Visual Domain Randomization for Reinforcement Learningcited by 13
- 2020Towards Interpretable-AI Policies Induction using Evolutionary Nonlinear Decision Trees for Discrete Action Systemscited by 12
- 2022Relational Abstractions for Generalized Reinforcement Learning on Symbolic Problemscited by 7
- 2022Learning Action Translator for Meta Reinforcement Learning on Sparse-Reward Taskscited by 5
- 2022Asynchronous Curriculum Experience Replay: A Deep Reinforcement Learning Approach for UAV Autonomous Motion Control in Unknown Dynamic Environmentscited by 4
- 2022Indoor Path Planning for Multiple Unmanned Aerial Vehicles via Curriculum Learningcited by 1
Built on by
Citing papers per year: 2025: 1.
1 paper.