mlsanityData to 14 Oct 2026

Learning from the Test: Self-Referential Differential Testing for Deep RL Agents

arXiv 2608.22284 ↗First posted 23 Aug 2026cs.SE cs.AI cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Safe offline RL

also in 1 more
Abstract (arXiv)

Deep Reinforcement Learning (DRL) has achieved significant success in complex decision-making problems. As DRL systems are increasingly deployed in real-world applications, ensuring their quality and reliability is paramount. Current works primarily focus on detecting safety-critical failures, often neglecting policy optimality, which can lead to reduced efficiency, user distrust, and economic losses. This oversight, compounded by the inherent "testing oracle problem" for optimality, leaves a significant gap in comprehensively evaluating DRL systems. To address this gap, we propose Delta (Differential Testing for DRL Agents), a novel and comprehensive framework that automatically identifies both safety-critical and optimality bugs in DRL agents. Delta employs a two-phase approach: (1) Safety Testing, where the Agent Under Test (AUT) is evaluated for catastrophic failures while collecting data from its decision-making policy, and (2) Optimality Testing, where this collected data from the prior phase is used to train a challenger agent via Offline Reinforcement Learning. Differential testing is then performed by comparing the challenger agent against the AUT; instances where the challenger achieves higher cumulative rewards indicate optimality issues in the AUT. We demonstrate Delta's effectiveness across five environments. We investigate the effectiveness of three offline RL algorithms (BC, BCQ, and CQL) in generating challenger agents. Experimental results demonstrate that safety testing datasets are valuable for training competent DRL agents. Challenger agents trained with BCQ proved most effective for identifying optimality issues within the framework of Delta. Across the five environments, Delta uncovered an average of 2,518 optimality issues, outperforming the baseline methods by 50.2%.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 50 of its 87 references that are in the map. First posted on arXiv 23 Aug 2026.

Builds on

50 of 87 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings23 Jul

Other references · 49

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  5. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  7. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  9. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,114
  10. 2018Exploration by Random Network Distillationcited by 993
  11. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  12. 2018Behavioral Cloning from Observationcited by 492
37 more
  1. 2017A Brief Survey of Deep Reinforcement Learningcited by 483
  2. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  3. 2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 311
  4. 2019Reinforcement Learning in Healthcare: A Surveycited by 310
  5. 2017Deep Reinforcement Learning framework for Autonomous Drivingcited by 225
  6. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 214
  7. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 213
  8. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 158
  9. 2024LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Aheadcited by 155
  10. 2018DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systemscited by 147
  11. 2018Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applicationscited by 127
  12. 2021d3rlpy: An Offline Deep Reinforcement Learning Librarycited by 84
  13. 2022Natural Attack for Pre-trained Models of Codecited by 75
  14. 2020Super-Human Performance in Gran Turismo Sport Using Deep Reinforcement Learningcited by 65
  15. 2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradientcited by 51
  16. 2019Making Efficient Use of Demonstrations to Solve Hard Exploration Problemscited by 45
  17. 2023Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcarecited by 42
  18. 2021A Survey on Locality Sensitive Hashing Algorithms and their Applicationscited by 40
  19. 2021Offline Reinforcement Learning for Autonomous Driving with Safety and Exploration Enhancementcited by 29
  20. 2022One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learningcited by 23
  21. 2022Many-Objective Reinforcement Learning for Online Testing of DNN-Enabled Systemscited by 19
  22. 2019Exploring applications of deep reinforcement learning for real-world autonomous driving systemscited by 18
  23. 2020Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluationcited by 14
  24. 2022Search-Based Testing of Reinforcement Learningcited by 13
  25. 2022A Search-Based Testing Approach for Deep Reinforcement Learning Agentscited by 13
  26. 2023Towards Assessing and Benchmarking Risk-Return Tradeoff of Off-Policy Evaluationcited by 13
  27. 2021MDPFuzz: Testing Models Solving Markov Decision Processescited by 12
  28. 2024muPRL: A Mutation Testing Pipeline for Deep Reinforcement Learning based on Real Faultscited by 5
  29. 2023Mutation Testing of Deep Reinforcement Learning Based on Real Faultscited by 4
  30. 2024Decictor: Towards Evaluating the Robustness of Decision-Making in Autonomous Driving Systemscited by 4
  31. 2025Scalable Decision-Making in Stochastic Environments through Learned Temporal Abstractioncited by 4
  32. 2025Curiosity-Driven Testing for Sequential Decision-Making Processcited by 3
  33. 2018Assurances in Software Testing: A Roadmapcited by 2
  34. 2023An Intentional Forgetting-Driven Self-Healing Method For Deep Reinforcement Learning Systemscited by 1
  35. 2024Finding Safety Violations of AI-Enabled Control Systems through the Lens of Synthesized Proxy Programscited by 1
  36. 2024Synthesizing Efficient and Permissive Programmatic Runtime Shields for Neural Policiescited by 1
  37. 2025Behavioral Entropy-Guided Dataset Generation for Offline Reinforcement Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Test Where Decisions Matter: Importance-driven Testing for Deep Reinforcement Learning
  2. 2026DreamTest: World-Model Surrogates for Search-Based Testing of Deep Reinforcement Learning Agents
  3. 2024Reinforcement Learning for Online Testing of Autonomous Driving Systems: a Replication and Extension Study
  4. 2025The Pursuit of Diversity: Multi-Objective Testing of Deep Reinforcement Learning Agents
  5. 2026FairTest: Search-Based Fairness Testing for Multi-Agent Reinforcement Learning Systems
  6. 2026Evaluating Fuzz Testing for Reinforcement Learning Agents
  7. 2024A Comprehensive Survey of Reinforcement Learning: From Algorithms to Practical Challenges
  8. 2025A Practical Introduction to Deep Reinforcement Learning