RefuteBench 2.0 -- Agentic Benchmark for Dynamic Evaluation of LLM Responses to Refutation Instruction
On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks
Abstract (arXiv)
In the multi-turn interaction schema, large language models (LLMs) can leverage user feedback to enhance the quality and relevance of their responses. However, evaluating an LLM's ability to incorporate user refutation feedback is crucial yet challenging. In this study, we introduce RefuteBench 2.0, which significantly extends the original RefuteBench by incorporating LLM agents as refuters and evaluators, which allows for flexible and comprehensive assessment. We design both transient and persistent refutation instructions with different validity periods. Meta-evaluation shows that the LLM-based refuter could generate more human-like refutations and the evaluators could assign scores with high correlation with humans. Experimental results of various LLMs show that current models could effectively satisfy the refutation but fail to memorize the refutation information. Interestingly, we also observe that the performance of the initial task decreases as the refutations increase. Analysis of the attention scores further shows a potential weakness of current LLMs: they struggle to retain and correctly use previous information during long context dialogues. https://github.com/ElliottYan/RefuteBench-2.0
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 39 of its 43 references that are in the map. First posted on arXiv 25 Feb 2025.
Builds on
39 of 43 references.
Closest ancestors · 1
Same direction, newest first
Other references · 38
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
- 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
- 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
- 2024Mixtral of Expertscited by 1,785
26 more
- 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
- 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
- 2018Hierarchical Neural Story Generationcited by 1,182
- 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
- 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
- 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
- 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
- 2023ChatGPT: Jack of all trades, master of nonecited by 204
- 2023MAGE: Machine-generated Text Detection in the Wildcited by 130
- 2016Winning Arguments: Interaction Dynamics and Persuasion Strategies in Good-faith Online Discussionscited by 106
- 2023NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classescited by 78
- 2023Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Modelscited by 48
- 2024A Multi-Agent Conversational Recommender Systemcited by 42
- 2023Revealing the structure of language model capabilitiescited by 35
- 2024Do LLM Agents Have Regret? A Case Study in Online Learning and Gamescited by 33
- 2023Never Lost in the Middle: Mastering Long-Context Question Answering with Position-Agnostic Decompositional Trainingcited by 31
- 2023Instruction-following Evaluation through Verbalizer Manipulationcited by 30
- 2024From Complex to Simple: Enhancing Multi-Constraint Complex Instruction Following Ability of Large Language Modelscited by 27
- 2021SciXGen: A Scientific Paper Dataset for Context-Aware Text Generationcited by 22
- 2023Towards Better Evaluation of Instruction-Following: A Case-Study in Summarizationcited by 18
- 2024The SIFo Benchmark: Investigating the Sequential Instruction Following Ability of Large Language Modelscited by 18
- 2025ThinkBench: Dynamic Out-of-Distribution Evaluation for Robust LLM Reasoningcited by 17
- 2024RefuteBench: Evaluating Refuting Instruction-Following for Large Language Modelscited by 10
- 2024PingPong: A Benchmark for Role-Playing Language Models with User Emulation and Multi-Model Evaluationcited by 10
- 2024Identifying Breakdowns in Conversational Recommender Systems using User Simulationcited by 4
- 2024Keys to Robust Edits: from Theoretical Insights to Practical Advancescited by 2
Built on by
Citing papers per year: 2025: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2025RefineBench: Evaluating Refinement Capability of Language Models via Checklists
- 2023Self-Refine: Iterative Refinement with Self-Feedback
- 2024CriticBench: Benchmarking LLMs for Critique-Correct Reasoning
- 2024FB-Bench: A Fine-Grained Multi-Task Benchmark for Evaluating LLMs' Responsiveness to Human Feedback
- 2023AgentBench: Evaluating LLMs as Agents
- 2025LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Research
- 2024Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate
- 2026MILE-RefHumEval: A Reference-Free, Multi-Independent LLM Framework for Human-Aligned Evaluation