mlsanityData to 7 Oct 2026

Towards Autonomous Testing Agents via Conversational Large Language Models

arXiv 2306.05152 ↗First posted 8 Jun 2023cs.SEcited by 13

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Software testing is an important part of the development cycle, yet it requires specialized expertise and substantial developer effort to adequately test software. Recent discoveries of the capabilities of large language models (LLMs) suggest that they can be used as automated testing assistants, and thus provide helpful information and even drive the testing process. To highlight the potential of this technology, we present a taxonomy of LLM-based testing agents based on their level of autonomy, and describe how a greater level of autonomy can benefit developers in practice. An example use of LLMs as a testing assistant is provided to demonstrate how a conversational framework for testing can help developers. This also highlights how the often criticized hallucination of LLMs can be beneficial for testing. We identify other tangible benefits that LLM-driven testing agents can bestow, and also discuss potential limitations.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 24 of its 37 references that are in the map. First posted on arXiv 8 Jun 2023.

Builds on

24 of 37 references.

References · 24

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  5. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  9. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  10. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  11. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  12. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
12 more
  1. 2022PAL: Program-aided Language Modelscited by 635
  2. 2018Model compression via distillation and quantizationcited by 287
  3. 2022Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproductioncited by 110
  4. 2023ChatGPT and Software Testing Education: Promises & Perilscited by 40
  5. 2023Explainable Automated Debugging via Large Language Model-driven Scientific Debuggingcited by 40
  6. 2017Continuous Integration, Delivery and Deployment: A Systematic Review on Approaches, Tools, Challenges and Practicescited by 22
  7. 2022An Analysis and Comparison of ACT-R and Soarcited by 12
  8. 2015Test Set Diameter: Quantifying the Diversity of Sets of Test Casescited by 10
  9. 2018Ways of Applying Artificial Intelligence in Software Engineeringcited by 7
  10. 2023A Qualitative Study on the Implementation Design Decisions of Developerscited by 7
  11. 2019An Experience Report On Applying Software Testing Academic Results In Industry: We Need Usable Automated Test Generationcited by 4
  12. 2020Boundary Value Exploration for Software Analysiscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 3; 2025: 4; 2026: 2.

13 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  3. 2024From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Futurecited by 85
  4. 2024Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communitiescited by 84
  5. 2023Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBenchcited by 54
  6. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Userscited by 32
  7. 2024Exploring Autonomous Agents through the Lens of Large Language Models: A Reviewcited by 30
  8. 2023Autonomous Large Language Model Agents Enabling Intent-Driven Mobile GUI Testingcited by 15
  9. 2025Large Language Models (LLMs) for Source Code Analysis: applications, models and datasetscited by 15
  10. 2025Automating a Complete Software Test Process Using LLMs: An Automotive Case Studycited by 8
  11. 2025COSMosFL: Ensemble of Small Language Models for Fault Localisationcited by 3
  12. 2026The Semi-Executable Stack: Agentic Software Engineering and the Expanding Scope of SEcited by 2
1 more
  1. 2026LLM-Driven CI-CD Workflow Intelligence for Cyber Systems Engineeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024The Potential of LLMs in Automating Software Testing: From Generation to Reporting
  2. 2026Multi-Agent LLM Collaboration for Unit Test Generation via Human-Testing-Inspired Workflows
  3. 2025Challenges in Testing Large Language Model Based Software: A Faceted Taxonomy
  4. 2026Tangent: An Empirical Study of Testing Practices for LLM-Based Agent Applications
  5. 2025Rethinking Testing for LLM Applications: Characteristics, Challenges, and a Lightweight Interaction Protocol
  6. 2026Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents
  7. 2024Creating an LLM-based AI-agent: A high-level methodology towards enhancing LLMs with APIs
  8. 2024Automated test generation to evaluate tool-augmented LLMs as conversational AI agents