mlsanityData to 14 Oct 2026

ToolHaystack: Stress-Testing Tool-Augmented Language Models in Realistic Long-Term Interactions

arXiv 2505.23662 ↗First posted 29 May 2025cs.CLcited by 5

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

Large language models (LLMs) have demonstrated strong capabilities in using external tools to address user inquiries. However, most existing evaluations assume tool use in short contexts, offering limited insight into model behavior during realistic long-term interactions. To fill this gap, we introduce ToolHaystack, a benchmark for testing the tool use capabilities in long-term interactions. Each test instance in ToolHaystack includes multiple tasks execution contexts and realistic noise within a continuous conversation, enabling assessment of how well models maintain context and handle various disruptions. By applying this benchmark to 14 state-of-the-art LLMs, we find that while current models perform well in standard multi-turn settings, they often significantly struggle in ToolHaystack, highlighting critical gaps in their long-term robustness not revealed by previous tool benchmarks.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 24 of its 60 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

24 of 60 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Benchmarking Failures in Tool-Augmented Language Models18 Mar
  2. 2025ToolDial: Multi-turn Dialogue Generation Method for Tool-Augmented Language Models1 Mar
  3. 2025ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling under Long-Context Scenario17 Jan
  4. 2024AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Calls6 Feb
  5. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs31 Jul
  6. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases8 Jun
  7. 2023Large Language Models as Tool Makers26 May
  8. 2023Gorilla: Large Language Model Connected with Massive APIs24 May
  9. 2023Toolformer: Language Models Can Teach Themselves to Use Tools9 Feb

Other references · 15

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  5. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  6. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 707
  7. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  8. 2025MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMscited by 180
  9. 2025APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplaycited by 167
  10. 2024Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigationcited by 126
  11. 2023T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Stepcited by 93
  12. 2024Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarioscited by 86
3 more
  1. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 79
  2. 2025Multi-Mission Tool Bench: Assessing the Robustness of LLM based Agents through Related and Dynamic Missionscited by 12
  3. 2024ToolBridge: An Open-Source Dataset to Equip LLMs with External Tool Capabilitiescited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  2. 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestrationcited by 14
  3. 2025Towards Reliable Benchmarking: A Contamination Free, Controllable Evaluation Framework for Multi-step LLM Function Callingcited by 13
  4. 2026Don't Blindly Trust It: How Unreliable Feedback Breaks Tool-Using LLM Agentscited by 3
  5. 2026Clueing up LLMs with Tool-Augmented Deductive Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Ideal Instruction: A Comprehensive Framework for Evaluating LLMs in Realistic Interactions
  2. 2026Benchmarking LLM Tool-Use in the Wild
  3. 2025LongFuncEval: Measuring the effectiveness of long context models for function calling
  4. 2024ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarios
  5. 2025ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Use
  6. 2026Lost in Execution: On the Multilingual Robustness of Tool Calling in Large Language Models
  7. 2025MCP-RADAR: A Multi-Dimensional Benchmark for Evaluating Tool Use Capabilities in Large Language Models
  8. 2024WTU-EVAL: A Whether-or-Not Tool Usage Evaluation Benchmark for Large Language Models