# Beyond Ideal Instruction: A Comprehensive Framework for Evaluating LLMs in Realistic Interactions

arXiv: https://arxiv.org/abs/2606.03318 · HTML: https://mlsanity.com/paper/2606.03318/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 23 references that are in the map.
- First posted on arXiv 2 Jun 2026.

## Builds on (most cited first, 12 of 12)

- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [GTA: A Benchmark for General Tool Agents](https://mlsanity.com/paper/2407.08713/) (2024)
- [ACEBench: Who Wins the Match Point in Tool Usage?](https://mlsanity.com/paper/2501.12851/) (2025)
- [Know You First and Be You Better: Modeling Human-Like User Simulators via Implicit Profiles](https://mlsanity.com/paper/2502.18968/) (2025)
- [Non-Collaborative User Simulators for Tool Agents](https://mlsanity.com/paper/2509.23124/) (2025)
- [Benchmarking LLM Tool-Use in the Wild](https://mlsanity.com/paper/2604.06185/) (2026)

## Built on by (most cited first, 1 of 1)

- [Asking for What Was Never Requested: Horizontal and Vertical Proactivity in Agents](https://mlsanity.com/paper/2609.37236/) (2026)

## Abstract (arXiv)

Despite great advances in tool-use capabilities of large language models (LLMs), existing evaluation benchmarks struggle to fully align with real-world scenarios. Such benchmarks mostly rely on simulated idealized user assumptions and lacks experience-oriented evaluation. These limitations fail to account for the ambiguity, uncooperative behaviors, and shifting intentions characteristic of real-world users. To fill this gap, we propose RUT-Bench, a dedicated benchmark designed to assess LLMs under diverse Real-world User Tool calling scenarios. RUT-Bench supports high-fidelity simulations covering both ideal rational patterns and heterogeneous non-ideal behaviors across single-turn and multi-turn dialogues. We conduct comprehensive evaluations on 19 widely adopted open-source and proprietary LLMs using our benchmark. Experimental results reveal that no tested LLMs achieve an overall success rate above 40%, and nearly all of them experience noticeable performance drops when facing more complicated non-ideal user inputs. Our code and data is available at https://github.com/Miaow-Lab/RUT-Bench.