mlsanityData to 7 Oct 2026

Testing and Understanding Erroneous Planning in LLM Agents through Synthesized User Inputs

arXiv 2404.17833 ↗First posted 27 Apr 2024cs.AI cs.PLcited by 22

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Agents based on large language models (LLMs) have demonstrated effectiveness in solving a wide range of tasks by integrating LLMs with key modules such as planning, memory, and tool usage. Increasingly, customers are adopting LLM agents across a variety of commercial applications critical to reliability, including support for mental well-being, chemical synthesis, and software development. Nevertheless, our observations and daily use of LLM agents indicate that they are prone to making erroneous plans, especially when the tasks are complex and require long-term planning. In this paper, we propose PDoctor, a novel and automated approach to testing LLM agents and understanding their erroneous planning. As the first work in this direction, we formulate the detection of erroneous planning as a constraint satisfiability problem: an LLM agent's plan is considered erroneous if its execution violates the constraints derived from the user inputs. To this end, PDoctor first defines a domain-specific language (DSL) for user queries and synthesizes varying inputs with the assistance of the Z3 constraint solver. These synthesized inputs are natural language paragraphs that specify the requirements for completing a series of tasks. Then, PDoctor derives constraints from these requirements to form a testing oracle. We evaluate PDoctor with three mainstream agent frameworks and two powerful LLMs (GPT-3.5 and GPT-4). The results show that PDoctor can effectively detect diverse errors in agent planning and provide insights and error characteristics that are valuable to both agent developers and users. We conclude by discussing potential alternative designs and directions to extend PDoctor.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 42 of its 62 references that are in the map. First posted on arXiv 27 Apr 2024.

Builds on

42 of 62 references.

References · 42

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  4. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  5. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  6. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  7. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  8. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  9. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  10. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  11. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  12. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 476
30 more
  1. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 372
  2. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 348
  3. 2023Character-LLM: A Trainable Agent for Role-Playingcited by 335
  4. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 331
  5. 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 329
  6. 2023On the Planning Abilities of Large Language Models : A Critical Investigationcited by 308
  7. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  8. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 268
  9. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 209
  10. 2023Is ChatGPT A Good Translator? Yes With GPT-4 As The Enginecited by 172
  11. 2023SmartPlay: A Benchmark for LLMs as Intelligent Agentscited by 123
  12. 2023SatLM: Satisfiability-Aided Language Models Using Declarative Promptingcited by 100
  13. 2024CriticBench: Benchmarking LLMs for Critique-Correct Reasoningcited by 88
  14. 2023Can large language models build causal graphs?cited by 72
  15. 2023Split and Merge: Aligning Position Biases in LLM-based Evaluatorscited by 71
  16. 2022A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Modelscited by 68
  17. 2023GPTScan: Detecting Logic Vulnerabilities in Smart Contracts by Combining GPT with Program Analysiscited by 65
  18. 2024LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs' Vulnerability Reasoningcited by 57
  19. 2023Testing Language Model Agents Safely in the Wildcited by 45
  20. 2024Combining Fine-Tuning and LLM-based Agents for Intuitive Smart Contract Auditing with Justificationscited by 41
  21. 2019Automatic Testing and Improvement of Machine Translationcited by 35
  22. 2022CCTEST: Testing and Repairing Code Completion Systemscited by 35
  23. 2024Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMscited by 28
  24. 2023Refining Decompiled C Code with Large Language Modelscited by 24
  25. 2023How susceptible are LLMs to Logical Fallacies?cited by 23
  26. 2021Efficient Combinatorial Optimization for Word-level Adversarial Textual Attackcited by 17
  27. 2023Symbol-LLM: Leverage Language Models for Symbolic System in Visual Human Activity Reasoningcited by 16
  28. 2023Benchmarking and Explaining Large Language Model-based Code Generation: A Causality-Centric Approachcited by 15
  29. 2023Interactive Joint Planning for Autonomous Vehiclescited by 13
  30. 2023Learning-based Near-optimal Motion Planning for Intelligent Vehicles with Uncertain Dynamicscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 14; 2026: 7.

22 papers.

Sort:

By area:

  1. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMscited by 424
  2. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  3. 2025Where LLM Agents Fail and How They can Learn From Failurescited by 118
  4. 2025OctoTools: An Agentic Framework with Extensible Tools for Complex Reasoningcited by 69
  5. 2025ACE: A Security Architecture for LLM-Integrated App Systemscited by 52
  6. 2025VeriPlan: Integrating Formal Verification and LLMs into End-User Planningcited by 19
  7. 2025SAMULE: Self-Learning Agents Enhanced by Multi-level Reflectioncited by 15
  8. 2026When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use Agentscited by 8
  9. 2026AgentForesight: Online Auditing for Early Failure Prediction in Multi-Agent Systemscited by 8
  10. 2025TAI3: Testing Agent Integrity in Interpreting User Intentcited by 7
  11. 2025A Good Plan is Hard to Find: Aligning Models with Preferences is Misaligned with What Helps Userscited by 6
  12. 2025Mobile-Agent-RAG: Driving Smart Multi-Agent Coordination with Contextual Knowledge Empowerment for Long-Horizon Mobile Automationcited by 5
10 more
  1. 2025Teaching AI to Handle Exceptions: Supervised Fine-Tuning with Human-Aligned Judgmentcited by 4
  2. 2025RAG or Fine-tuning? A Comparative Study on LCMs-based Code Completion in Industrycited by 4
  3. 2025Automated Penetration Testing with LLM Agents and Classical Planningcited by 4
  4. 2026PIVOT: Bridging Planning and Execution in LLM Agents via Trajectory Refinementcited by 4
  5. 2025Agent WARPP: Workflow Adherence via Runtime Parallel Personalizationcited by 3
  6. 2025TREAT: A Code LLMs Trustworthiness / Reliability Evaluation and Testing Frameworkcited by 1
  7. 2026Assessing Deanonymization Risks with Stylometry-Assisted LLM Agentcited by 1
  8. 2026Mosaic: Runtime-Efficient Multi-Agent Embodied Planningcited by 1
  9. 2026HPFA: Hypergraph-Based Paired Failure Attribution for LLM Reasoningcited by 1
  10. 2026On the Suitability of LLM-Driven Agents for Dark Pattern Auditscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LLM Reasoner and Automated Planner: A new NPC approach
  2. 2024Can We Rely on LLM Agents to Draft Long-Horizon Plans? Let's Take TravelPlanner as an Example
  3. 2024Formal-LLM: Integrating Formal Language and Natural Language for Controllable LLM-based Agents
  4. 2025Scaling LLM Planning: NL2FLOW for Parametric Problem Generation and Rigorous Evaluation
  5. 2023On the Planning Abilities of Large Language Models (A Critical Investigation with a Proposed Benchmark)
  6. 2026Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agents
  7. 2026PDDLCoder: Agentic PDDL Generation for LLM-Assisted Symbolic Planning
  8. 2026Training the Orchestrator: A Supervised Approach to End-to-End PDDL Planning with LLM Agents