mlsanityData to 14 Oct 2026

On the Mistaken Assumption of Interchangeable Deep Reinforcement Learning Implementations

arXiv 2503.22575 ↗First posted 28 Mar 2025cs.SE cs.AIcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Deep Reinforcement Learning (DRL) is a paradigm of artificial intelligence where an agent uses a neural network to learn which actions to take in a given environment. DRL has recently gained traction from being able to solve complex environments like driving simulators, 3D robotic control, and multiplayer-online-battle-arena video games. Numerous implementations of the state-of-the-art algorithms responsible for training these agents, like the Deep Q-Network (DQN) and Proximal Policy Optimization (PPO) algorithms, currently exist. However, studies make the mistake of assuming implementations of the same algorithm to be consistent and thus, interchangeable. In this paper, through a differential testing lens, we present the results of studying the extent of implementation inconsistencies, their effect on the implementations' performance, as well as their impact on the conclusions of prior studies under the assumption of interchangeable implementations. The outcomes of our differential tests showed significant discrepancies between the tested algorithm implementations, indicating that they are not interchangeable. In particular, out of the five PPO implementations tested on 56 games, three implementations achieved superhuman performance for 50% of their total trials while the other two implementations only achieved superhuman performance for less than 15% of their total trials. As part of a meticulous manual analysis of the implementations' source code, we analyzed implementation discrepancies and determined that code-level inconsistencies primarily caused these discrepancies. Lastly, we replicated a study and showed that this assumption of implementation interchangeability was sufficient to flip experiment outcomes. Therefore, this calls for a shift in how implementations are being used.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 41 of its 81 references that are in the map. First posted on arXiv 28 Mar 2025.

Builds on

41 of 81 references.

References · 41

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  6. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  7. 2016OpenAI Gymcited by 1,736
  8. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  9. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  10. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  11. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
  12. 2017Deep Reinforcement Learning that Matterscited by 754
29 more
  1. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  2. 2017DeepXplore: Automated Whitebox Testing of Deep Learning Systemscited by 311
  3. 2021MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learningcited by 290
  4. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  5. 2021Accounting for Variance in Machine Learning Benchmarkscited by 201
  6. 2019Soft Actor-Critic for Discrete Action Settingscited by 174
  7. 2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 143
  8. 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 122
  9. 2022Large Language Models are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Modelscited by 103
  10. 2017Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Controlcited by 93
  11. 2023Fuzz4All: Universal Fuzzing with Large Language Modelscited by 88
  12. 2018DLFuzz: Differential Fuzzing Testing of Deep Learning Systemscited by 65
  13. 2021FinRL: Deep Reinforcement Learning Framework to Automate Trading in Quantitative Financecited by 58
  14. 2022Atari-5: Distilling the Arcade Learning Environment down to Five Gamescited by 44
  15. 2020Model-based Exploration of the Frontier of Behaviours for Deep Learning System Testingcited by 40
  16. 2023WhiteFox: White-Box Compiler Fuzzing Empowered by Large Language Modelscited by 37
  17. 2022Free Lunch for Testing: Fuzzing Deep-Learning Libraries from Open Sourcecited by 28
  18. 2022Toward Understanding Deep Learning Framework Bugscited by 26
  19. 2022Beyond CAGE: Investigating Generalization of Learned Autonomous Network Defense Policiescited by 23
  20. 2023Latent Exploration for Reinforcement Learningcited by 22
  21. 2023Dynamics Generalisation in Reinforcement Learning via Adaptive Context-Aware Policiescited by 20
  22. 2023KernelGPT: Enhanced Kernel Fuzzing via Large Language Modelscited by 17
  23. 2022Fuzzing Deep-Learning Libraries via Automated Relational API Inferencecited by 16
  24. 2022Differential testing for machine learning: an analysis for classification algorithms beyond deep learningcited by 3
  25. 2022A Comparison of Reinforcement Learning Frameworks for Software Testing Taskscited by 3
  26. 2023DeltaNN: Assessing the Impact of Computational Environment Parameters on the Performance of Image Recognition Modelscited by 3
  27. 2023Characterizing Bugs in Python and R Data Analytics Programscited by 3
  28. 2023Selectively Sharing Experiences Improves Multi-Agent Reinforcement Learningcited by 3
  29. 2023Review of Deep Reinforcement Learning for Autonomous Drivingcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025A Practical Introduction to Deep Reinforcement Learning
  2. 2025A Survey on Explainable Deep Reinforcement Learning
  3. 2026Combining Trained Models in Reinforcement Learning
  4. 2025Control-Optimized Deep Reinforcement Learning for Artificially Intelligent Autonomous Systems
  5. 2024Efficient Exploration in Deep Reinforcement Learning: A Novel Bayesian Actor-Critic Algorithm
  6. 2026Deep Reinforcement Learning and The Tale of Two Temporal Difference Errors
  7. 2024Towards Adapting Reinforcement Learning Agents to New Tasks: Insights from Q-Values
  8. 2024A Comprehensive Survey of Reinforcement Learning: From Algorithms to Practical Challenges