mlsanityData to 14 Oct 2026

Robustness Evaluation of Offline Reinforcement Learning for Robot Control Against Action Perturbations

arXiv 2412.18781 ↗First posted 25 Dec 2024cs.RO cs.LGcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning, which learns solely from datasets without environmental interaction, has gained attention. This approach, similar to traditional online deep reinforcement learning, is particularly promising for robot control applications. Nevertheless, its robustness against real-world challenges, such as joint actuator faults in robots, remains a critical concern. This study evaluates the robustness of existing offline reinforcement learning methods using legged robots from OpenAI Gym based on average episodic rewards. For robustness evaluation, we simulate failures by incorporating both random and adversarial perturbations, representing worst-case scenarios, into the joint torque signals. Our experiments show that existing offline reinforcement learning methods exhibit significant vulnerabilities to these action perturbations and are more vulnerable than online reinforcement learning methods, highlighting the need for more robust approaches in this field.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 51 of its 68 references that are in the map. First posted on arXiv 25 Dec 2024.

Builds on

51 of 68 references.

Closest ancestors · 24

Same direction, newest first

  1. 2024Is Value Learning Really the Main Bottleneck in Offline RL?13 Jun
  2. 2024Towards Robust Policy: Enhancing Offline Reinforcement Learning with Adversarial Attacks and Defenses18 May
  3. 2023The Generalization Gap in Offline Reinforcement Learning10 Dec
  4. 2023MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operator7 Dec
  5. 2023Corruption-Robust Offline Reinforcement Learning with General Function Approximation23 Oct
  6. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruption19 Oct
  7. 2023Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coverage16 May
  8. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learning14 Mar
  9. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies15 Dec
  10. 2022Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Size20 Nov
  11. 2022BAFFLE: Hiding Backdoors in Offline Reinforcement Learning Datasets7 Oct
  12. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity11 Aug
12 more
  1. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing6 Jun
  2. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  3. 2021d3rlpy: An Offline Deep Reinforcement Learning Library6 Nov
  4. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  5. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  6. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  7. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  8. 2021Corruption-Robust Offline Reinforcement Learning11 Jun
  9. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  10. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  11. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  12. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 27

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  7. 2017Hindsight Experience Replaycited by 1,136
  8. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  9. 2017Robust Adversarial Reinforcement Learningcited by 451
  10. 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 238
  11. 2018Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experiencecited by 217
  12. 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 174
15 more
  1. 2022Robust Reinforcement Learning using Offline Datacited by 96
  2. 2021Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RLcited by 55
  3. 2021DeepThermal: Combustion Optimization for Thermal Power Generating Units Using Offline Reinforcement Learningcited by 48
  4. 2021Robust Policy Gradient against Strong Data Corruptioncited by 30
  5. 2023Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shapingcited by 25
  6. 2022Offline Reinforcement Learning for Safer Blood Glucose Control in People with Type 1 Diabetescited by 23
  7. 2023Benchmarking Offline Reinforcement Learning on Real-Robot Hardwarecited by 23
  8. 2023Distributionally Robust Model-based Reinforcement Learning with Large State Spacescited by 17
  9. 2022Provable Sim-to-real Transfer in Continuous Domain with Partial Observationscited by 9
  10. 2024Robust Deep Reinforcement Learning with Adaptive Adversarial Perturbations in Action Spacecited by 9
  11. 2022Cyclic Policy Distillation: Sample-Efficient Sim-to-Real Reinforcement Learning with Domain Randomizationcited by 8
  12. 2022Adversarial joint attacks on legged robotscited by 4
  13. 2024ROLeR: Effective Reward Shaping in Offline Reinforcement Learning for Recommender Systemscited by 4
  14. 2023Toward Evaluating Robustness of Reinforcement Learning with Adversarial Policycited by 3
  15. 2020Learning Task-Agnostic Action Spaces for Movement Optimizationcited by 1

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Adversarial Fine-tuning in Offline-to-Online Reinforcement Learning for Robust Robot Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Rethinking Robustness Assessment: Adversarial Attacks on Learning-based Quadrupedal Locomotion Controllers
  2. 2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learning
  3. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning
  4. 2023A Real-World Quadrupedal Locomotion Benchmark for Offline Reinforcement Learning
  5. 2023Robust Offline Reinforcement Learning -- Certify the Confidence Interval
  6. 2025Uncertainty-Aware Robotic World Model Makes Offline Model-Based Reinforcement Learning Work on Real Robots
  7. 2024Improving Offline Reinforcement Learning with Inaccurate Simulators
  8. 2024LiRA: Light-Robust Adversary for Model-based Reinforcement Learning in Real World