Robust Recurrent Reinforcement Learning under Evolving Hidden Disturbances with Application to Rover Wheel Slip
Not on the map. Why?
Abstract (arXiv)
Reinforcement learning (RL) performs well in continuous-control tasks, but evolving hidden disturbances create partial observability: the agent must infer decision-relevant latent dynamics from interaction history. This study investigates how observation history, action history, history length, and network structure affect recurrent Twin Delayed Deep Deterministic Policy Gradient (TD3) agents. Three recurrent architectures are evaluated under controlled disturbances with different temporal characteristics. Results show that action history is particularly important when observed responses depend on previous actions, and that processing past and current action-observation information within a unified temporal sequence improves performance compared with using separate branches. We also introduce H-TD3, which reuses recurrent states generated by the actor to initialize the critic, reducing duplicated sequence processing. The architectures are further tested in a simulation-based differential-drive rover motion-regulation task under hidden asymmetric wheel slip. Recurrent architectures retain their advantage under the physically motivated multiplicative wheel-slip model, while policies trained with abstract temporally structured disturbances transfer more effectively to previously unseen wheel-slip dynamics than policies trained without disturbances. These findings provide practical guidance for recurrent RL under partial observability and evolving hidden disturbances.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 29 of its 41 references that are in the map. First posted on arXiv 29 Jul 2023.
Builds on
29 of 41 references.
References · 29
- 2015Continuous control with deep reinforcement learningcited by 3,279
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
- 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
- 2016OpenAI Gymcited by 1,684
- 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
- 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
- 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
- 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
- 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
- 2022Learning robust perceptive locomotion for quadrupedal robots in the wildcited by 299
- 2018A Survey and Critique of Multiagent Deep Reinforcement Learningcited by 173
- 2020Liquid Time-constant Networkscited by 160
17 more
- 2020iGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenescited by 151
- 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 115
- 2018Motion Planning Among Dynamic, Decision-Making Agents with Deep Reinforcement Learningcited by 101
- 2015Memory-based control with recurrent neural networkscited by 77
- 2020A Survey of Reinforcement Learning Algorithms for Dynamically Varying Environmentscited by 62
- 2021Memory-based Deep Reinforcement Learning for POMDPscited by 53
- 2018On Improving Deep Reinforcement Learning for POMDPscited by 49
- 2021Partially Observable Markov Decision Processes (POMDPs) and Roboticscited by 49
- 2021Causal Reinforcement Learning using Observational and Interventional Datacited by 38
- 2022Rethinking Sim2Real: Lower Fidelity Simulation Leads to Higher Sim2Real Transfer in Navigationcited by 28
- 2018Memory-based Deep Reinforcement Learning for Obstacle Avoidance in UAV with Limited Environment Knowledgecited by 21
- 2021Recurrent Off-policy Baselines for Memory-based Continuous Controlcited by 19
- 2015Recurrent Reinforcement Learning: A Hybrid Approachcited by 13
- 2022Dynamics-Adaptive Continual Reinforcement Learning via Progressive Contextualizationcited by 13
- 2019Adaptive Guidance with Reinforcement Meta-Learningcited by 2
- 2022Deep reinforcement learning oriented for real world dynamic scenarioscited by 1
- 2022On Separation Between Learning and Control in Partially Observed Markov Decision Processescited by 1
Built on by
Citing papers per year: 2025: 1.
1 paper.