KC-Bench: A Dynamic Interactive Benchmark for Evaluating Knowledge Conflicts in LLM Agents
On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks
Abstract (arXiv)
As LLMs increasingly act through tools, they must reconcile user instructions, parametric knowledge, and dynamic environmental observations before taking actions. We introduce KC-Bench, a controlled multi-turn benchmark for measuring this capability across world-knowledge conflicts, input inconsistencies, and multi-source temporal conflicts. Its 238 tasks are manually screened from more than 1,000 generated candidates and combine a user simulator, stateful tools, deterministic environment assertions, an open-source natural-language evaluator, and human trajectory verification. Evaluation of nine models, including DeepSeek-V4-Flash, GLM-5.2, and MiniMax-M3, shows substantial cross-domain variation: no model handles factual correction, identity consistency checking, and temporal conflict resolution reliably across all settings. In the simulated environments, missed conflicts can propagate to tool calls or synthetic protected-data flows. KC-Bench isolates this model-level behavior rather than ranking complete agent frameworks, and provides a reproducible diagnostic for developing conflict-aware reasoning and execution safeguards.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 19 of its 26 references that are in the map. First posted on arXiv 3 Sep 2026.
Builds on
19 of 26 references.
Closest ancestors · 4
Same direction, newest first
- 2025Large Language Model Agent: A Survey on Methodology, Applications and Challenges27 Mar
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains17 Jun
- 2024A Review of Prominent Paradigms for LLM-Based Agents: Tool Use (Including RAG), Planning, and Feedback Learning9 Jun
- 2023A Survey on Large Language Model based Autonomous Agents22 Aug
Other references · 15
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
- 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 627
- 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 341
- 2024Knowledge Conflicts for LLMs: A Surveycited by 283
- 2025AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agentscited by 223
- 2025Prompt Injection Attack to Tool Selection in LLM Agentscited by 158
- 2022Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidencecited by 144
- 2024Tug-of-War Between Knowledge: Exploring and Resolving Knowledge Conflicts in Retrieval-Augmented Language Modelscited by 93
- 2024ConflictBank: A Benchmark for Evaluating the Influence of Knowledge Conflicts in LLMcited by 73
- 2024Blinded by Generated Contexts: How Language Models Merge Generated and Retrieved Contexts When Knowledge Conflicts?cited by 48
- 2023Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generatorscited by 43
- 2025Web-Bench: A LLM Code Benchmark Based on Web Standards and Frameworkscited by 40
3 more
- 2026Towards Verifiably Safe Tool Use for LLM Agentscited by 36
- 2025From LLMs to MLLMs to Agents: A Survey of Emerging Paradigms in Jailbreak Attacks and Defenses within LLM Ecosystemcited by 12
- 2025GhostEI-Bench: Do Mobile Agents Resilience to Environmental Injection in Dynamic On-Device Environments?cited by 12
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026AgentEscapeBench: Evaluating Out-of-Domain Tool-Grounded Reasoning in LLM Agents
- 2025Diagnose, Localize, Align: A Full-Stack Framework for Reliable LLM Multi-Agent Systems under Instruction Conflicts
- 2026LatticeMind: A Conflict-Aware Memory Primitive for Multi-Agent Systems
- 2026MANTRA: Synthesizing SMT-Validated Compliance Benchmarks for Tool-Using LLM Agents
- 2024AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents
- 2025Multi-Mission Tool Bench: Assessing the Robustness of LLM based Agents through Related and Dynamic Missions
- 2023AgentBench: Evaluating LLMs as Agents
- 2026VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks