mlsanityData to 14 Oct 2026

Learning From Failure: Integrating Negative Examples when Fine-tuning Large Language Models as Agents

arXiv 2402.11651 ↗First posted 18 Feb 2024cs.CLcited by 50

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Large language models (LLMs) have achieved success in acting as agents, which interact with environments through tools such as search engines. However, LLMs are optimized for language generation instead of tool use during training or alignment, limiting their effectiveness as agents. To resolve this problem, previous work has first collected interaction trajectories between LLMs and environments, using only trajectories that successfully finished the task to fine-tune smaller models, making fine-tuning data scarce and acquiring it both difficult and costly. Discarding failed trajectories also leads to significant wastage of data and resources and limits the possible optimization paths during fine-tuning. In this paper, we argue that unsuccessful trajectories offer valuable insights, and LLMs can learn from these trajectories through appropriate quality control and fine-tuning strategies. By simply adding a prefix or suffix that tells the model whether to generate a successful trajectory during training, we improve model performance by a large margin on mathematical reasoning, multi-hop question answering, and strategic question answering tasks. We further analyze the inference results and find that our method provides a better trade-off between valuable information and errors in unsuccessful trajectories. To our knowledge, we are the first to demonstrate the value of negative trajectories and their application in agent-tunning scenarios. Our findings offer guidance for developing better agent-tuning methods and low-resource data usage techniques.

Key facts. Cited by 50 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 21 of its 40 references that are in the map. First posted on arXiv 18 Feb 2024.

Builds on

21 of 40 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMs19 Oct
  2. 2023FireAct: Toward Language Agent Fine-tuning9 Oct
  3. 2023ExpeL: LLM Agents Are Experiential Learners20 Aug

Other references · 18

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  9. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  10. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  11. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  12. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
6 more
  1. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  2. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
  3. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 548
  4. 2016Solving General Arithmetic Word Problemscited by 360
  5. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 318
  6. 2024AutoAct: Automatic Agent Learning from Scratch for QA via Self-Planningcited by 81

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 13; 2025: 21; 2026: 16.

50 papers.

Sort:

By area:

  1. 2024AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generationcited by 83
  2. 2024ToolGen: Unified Tool Retrieval and Calling via Generationcited by 74
  3. 2024Agent Planning with World Knowledge Modelcited by 70
  4. 2024KnowAgent: Knowledge-Augmented Planning for LLM-Based Agentscited by 61
  5. 2025A Survey of Large Language Model Agents for Question Answeringcited by 48
  6. 2025A Survey on the Optimization of Large Language Model-based Agentscited by 46
  7. 2025Infrastructure for AI Agentscited by 41
  8. 2024BUMBLE: Unifying Reasoning and Acting with Vision-Language Models for Building-wide Mobile Manipulationcited by 32
  9. 2024DynaSaur: Large Language Agents Beyond Predefined Actionscited by 32
  10. 2025AgentRM: Enhancing Agent Generalization with Reward Modelingcited by 27
  11. 2025Stay Focused: Problem Drift in Multi-Agent Debatecited by 23
  12. 2024Can We Rely on LLM Agents to Draft Long-Horizon Plans? Let's Take TravelPlanner as an Examplecited by 22
38 more
  1. 2024From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learningcited by 22
  2. 2025Training LLM-Based Agents with Synthetic Self-Reflected Trajectories and Partial Maskingcited by 17
  3. 2026SpanVLA: Efficient Action Bridging and Learning from Negative-Recovery Samples for Vision-Language-Action Modelcited by 17
  4. 2025SAMULE: Self-Learning Agents Enhanced by Multi-level Reflectioncited by 15
  5. 2025ReCode: Unify Plan and Action for Universal Granularity Controlcited by 15
  6. 2026BAGEN: Are LLM Agents Budget-Aware?cited by 13
  7. 2024E2CL: Exploration-based Error Correction Learning for Embodied Agentscited by 9
  8. 2025Exploring Expert Failures Improves LLM Agent Tuningcited by 9
  9. 2025Unleashing Embodied Task Planning Ability in LLMs via Reinforcement Learningcited by 9
  10. 2024Table-LLM-Specialist: Language Model Specialists for Tables using Iterative Generator-Validator Fine-tuningcited by 8
  11. 2025InstructRAG: Leveraging Retrieval-Augmented Generation on Instruction Graphs for LLM-Based Task Planningcited by 8
  12. 2025PGPO: Enhancing Agent Reasoning via Pseudocode-style Planning Guided Preference Optimizationcited by 8
  13. 2026On-Policy Self-Evolution via Failure Trajectories for Agentic Safety Alignmentcited by 8
  14. 2026Unveiling Implicit Advantage Symmetry: Why GRPO Struggles with Exploration and Difficulty Adaptationcited by 7
  15. 2026Retrieval-Augmented LLM Agents: Learning to Learn from Experiencecited by 7
  16. 2025Hephaestus: Improving Fundamental Agent Capabilities of Large Language Models through Continual Pre-Trainingcited by 6
  17. 2025Unearthing Gems from Stones: Policy Optimization with Negative Sample Augmentation for LLM Reasoningcited by 6
  18. 2025PLAN-TUNING: Post-Training Language Models to Learn Step-by-Step Planning for Complex Problem Solvingcited by 6
  19. 2026Long-Chain Reasoning Distillation via Adaptive Prefix Alignmentcited by 6
  20. 2024Retrieved In-Context Principles from Previous Mistakescited by 5
  21. 2026Learning from Mistakes: Negative Reasoning Samples Enhance Out-of-Domain Generalizationcited by 5
  22. 2025Tru-POMDP: Task Planning Under Uncertainty via Tree of Hypotheses and Open-Ended POMDPscited by 4
  23. 2025Solving the Granularity Mismatch: Hierarchical Preference Learning for Long-Horizon LLM Agentscited by 4
  24. 2025Refining Positive and Toxic Samples for Dual Safety Self-Alignment of LLMs with Minimal Human Interventionscited by 3
  25. 2025Enhancing Long-Chain Reasoning Distillation through Error-Aware Self-Reflectioncited by 3
  26. 2026Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradientscited by 3
  27. 2026Constraint Tax in Open-Weight LLMs: An Empirical Study of Tool Calling Suppression Under Structured Output Constraintscited by 3
  28. 2026Diagnosis Before Recovery: Turning Agent Failures into Selective Self-Correctioncited by 3
  29. 2025Divide and Conquer: Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learningcited by 2
  30. 2024Associative memory and dead neuronscited by 1
  31. 2025Towards Safer Chatbots: Automated Policy Compliance Evaluation of Custom GPTscited by 1
  32. 2026Reversa: A Reverse Documentation Engineering Framework for Converting Legacy Software into Operational Specifications for AI Agentscited by 1
  33. 2026CurateEvo: Data-Curation Evolving for Agentic Post-Trainingcited by 1
  34. 2026Wrong but Useful: Trajectory Value Beyond Answer Correctness in Multi-Agent Messagescited by 1
  35. 2024Towards Adaptive Mechanism Activation in Language Agentcited by 0
  36. 2026Not All Negative Samples Are Equal: LLMs Learn Better from Plausible Reasoningcited by 0
  37. 2026Learning from Reliable Negatives: Confidence-Anchored Test-Time Adaptation for GUI Groundingcited by 0
  38. 2026PivotOPD: Learning to Recover from Pivotal Mistakes in Multi-Turn Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Models
  2. 2026Search, Do not Guess: Teaching Small Language Models to Be Effective Search Agents
  3. 2026Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Study
  4. 2025ATLaS: Agent Tuning via Learning Critical Steps
  5. 2026Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agents
  6. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debate
  7. 2025A Survey of Scaling in Large Language Model Reasoning
  8. 2025Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Tasks