# Towards Autonomous Testing Agents via Conversational Large Language Models arXiv: https://arxiv.org/abs/2306.05152 ยท HTML: https://mlsanity.com/paper/2306.05152/ ## Key facts - Cited by 13 papers in the map (data through 14 Oct 2026). - Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. - Builds on 24 of its 37 references that are in the map. - First posted on arXiv 8 Jun 2023. ## Placed in - LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/) ## Builds on (most cited first, 24 of 24) - [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021) - [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022) - [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020) - [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023) - [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023) - [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022) - [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023) - [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020) - [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023) - [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022) - [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://mlsanity.com/paper/2303.17580/) (2023) - [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023) - [PAL: Program-aided Language Models](https://mlsanity.com/paper/2211.10435/) (2022) - [Model compression via distillation and quantization](https://mlsanity.com/paper/1802.05668/) (2018) - [Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction](https://mlsanity.com/paper/2209.11515/) (2022) - [ChatGPT and Software Testing Education: Promises & Perils](https://mlsanity.com/paper/2302.03287/) (2023) - [Explainable Automated Debugging via Large Language Model-driven Scientific Debugging](https://mlsanity.com/paper/2304.02195/) (2023) - [Continuous Integration, Delivery and Deployment: A Systematic Review on Approaches, Tools, Challenges and Practices](https://mlsanity.com/paper/1703.07019/) (2017) - [An Analysis and Comparison of ACT-R and Soar](https://mlsanity.com/paper/2201.09305/) (2022) - [Test Set Diameter: Quantifying the Diversity of Sets of Test Cases](https://mlsanity.com/paper/1506.03482/) (2015) - [Ways of Applying Artificial Intelligence in Software Engineering](https://mlsanity.com/paper/1802.02033/) (2018) - [A Qualitative Study on the Implementation Design Decisions of Developers](https://mlsanity.com/paper/2301.09789/) (2023) - [An Experience Report On Applying Software Testing Academic Results In Industry: We Need Usable Automated Test Generation](https://mlsanity.com/paper/1901.03865/) (2019) - [Boundary Value Exploration for Software Analysis](https://mlsanity.com/paper/2001.06652/) (2020) ## Built on by (most cited first, 13 of 13) - [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023) - [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023) - [Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communities](https://mlsanity.com/paper/2407.07791/) (2024) - [From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future](https://mlsanity.com/paper/2408.02479/) (2024) - [Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBench](https://mlsanity.com/paper/2308.03656/) (2023) - [From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Users](https://mlsanity.com/paper/2508.17281/) (2025) - [Exploring Autonomous Agents through the Lens of Large Language Models: A Review](https://mlsanity.com/paper/2404.04442/) (2024) - [Large Language Models (LLMs) for Source Code Analysis: applications, models and datasets](https://mlsanity.com/paper/2503.17502/) (2025) - [Autonomous Large Language Model Agents Enabling Intent-Driven Mobile GUI Testing](https://mlsanity.com/paper/2311.08649/) (2023) - [Automating a Complete Software Test Process Using LLMs: An Automotive Case Study](https://mlsanity.com/paper/2502.04008/) (2025) - [COSMosFL: Ensemble of Small Language Models for Fault Localisation](https://mlsanity.com/paper/2502.02908/) (2025) - [The Semi-Executable Stack: Agentic Software Engineering and the Expanding Scope of SE](https://mlsanity.com/paper/2604.15468/) (2026) - [LLM-Driven CI-CD Workflow Intelligence for Cyber Systems Engineering](https://mlsanity.com/paper/2607.04579/) (2026) ## Abstract (arXiv) Software testing is an important part of the development cycle, yet it requires specialized expertise and substantial developer effort to adequately test software. Recent discoveries of the capabilities of large language models (LLMs) suggest that they can be used as automated testing assistants, and thus provide helpful information and even drive the testing process. To highlight the potential of this technology, we present a taxonomy of LLM-based testing agents based on their level of autonomy, and describe how a greater level of autonomy can benefit developers in practice. An example use of LLMs as a testing assistant is provided to demonstrate how a conversational framework for testing can help developers. This also highlights how the often criticized hallucination of LLMs can be beneficial for testing. We identify other tangible benefits that LLM-driven testing agents can bestow, and also discuss potential limitations.