mlsanityData to 7 Oct 2026

Gimitest: A Comprehensive Tool for Testing Reinforcement Learning Policies

arXiv 2607.07029 ↗First posted 8 Jul 2026cs.LG cs.AI cs.SEcited by 0

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning (RL) policies can be unsafe and vulnerable to attacks. Ensuring their reliability is often a pain point as existing automated testing methods target only selected environments, testing scenarios, and RL algorithms. To address this, we propose a comprehensive framework for testing single- and multi-agent RL policies under varying conditions. Our implementation of this framework, Gimitest, is an open-source tool that supports various gym frameworks and allows for modifications of their integrated components. This article describes the framework and details Gimitest's functionality and architecture. It showcases its effectiveness in testing multiple RL policies in environments such as the official Farama Gymnasium and PettingZoo.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 36 of its 68 references that are in the map. First posted on arXiv 8 Jul 2026.

Builds on

36 of 68 references.

References · 36

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  3. 2013Intriguing properties of neural networkscited by 5,114
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  6. 2017Adversarial Attacks on Neural Network Policiescited by 314
  7. 2017DeepXplore: Automated Whitebox Testing of Deep Learning Systemscited by 307
  8. 2017DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Carscited by 249
  9. 2020PettingZoo: Gym for Multi-Agent Reinforcement Learningcited by 209
  10. 2023An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generationcited by 193
  11. 2017Tactics of Adversarial Attack on Deep Reinforcement Learning Agentscited by 163
  12. 2023Software Testing with Large Language Models: Survey, Landscape, and Visioncited by 163
24 more
  1. 2017Delving into adversarial attacks on deep policiescited by 104
  2. 2022A Survey of Multi-Agent Deep Reinforcement Learning with Communicationcited by 103
  3. 2021Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Potcited by 97
  4. 2020Metamorphic Testing: A New Approach for Generating Next Test Casescited by 71
  5. 2020Stealthy and Efficient Adversarial Attacks against Deep Reinforcement Learningcited by 62
  6. 2023Effective Test Generation Using Pre-trained Large Language Models and Mutation Testingcited by 51
  7. 2022Deep Surrogate Assisted Generation of Environmentscited by 40
  8. 2022Code Generation Tools (Almost) for Free? A Study of Few-Shot, Pre-Trained Language Models on Codecited by 39
  9. 2022COOL-MC: A Comprehensive Tool for Reinforcement Learning and Model Checkingcited by 19
  10. 2022Many-Objective Reinforcement Learning for Online Testing of DNN-Enabled Systemscited by 18
  11. 2019MinAtar: An Atari-Inspired Testbed for Thorough and Reproducible Reinforcement Learning Experimentscited by 17
  12. 2022Search-Based Testing of Reinforcement Learningcited by 12
  13. 2022A Search-Based Testing Approach for Deep Reinforcement Learning Agentscited by 12
  14. 2023Testing of Deep Reinforcement Learning Agents with Surrogate Modelscited by 12
  15. 2021MDPFuzz: Testing Models Solving Markov Decision Processescited by 11
  16. 2019Adaptive Metamorphic Testing with Contextual Banditscited by 7
  17. 2023Robustness Testing for Multi-Agent Reinforcement Learning: State Perturbations on Critical Agentscited by 7
  18. 2021Mungojerrie: Reinforcement Learning of Linear-Time Objectivescited by 6
  19. 2022Targeted Adversarial Attacks on Deep Reinforcement Learning Policies via Model Checkingcited by 5
  20. 2024Evaluating Human Trajectory Prediction with Metamorphic Testingcited by 3
  21. 2025Policy Testing with MDPFuzz (Replicability Study)cited by 3
  22. 2024Testing for Fault Diversity in Reinforcement Learningcited by 2
  23. 2022Artificial Intelligence in Software Testing : Impact, Problems, Challenges and Prospectcited by 1
  24. 2023A Review of Validation and Verification of Neural Network-based Policies for Sequential Decision Makingcited by 1

Built on by

Not cited yet.