Feedback Indices to Evaluate LLM Responses to Rebuttals for Multiple Choice Type Questions
Not on the map. Why?
Abstract (arXiv)
We present a systematic framework of indices designed to characterize Large Language Model (LLM) responses when challenged with rebuttals during a chat. Assessing how LLMs respond to user dissent is crucial for understanding their reliability and behavior patterns, yet the complexity of human-LLM interactions makes systematic evaluation challenging. Our approach employs a fictitious-response rebuttal method that quantifies LLM behavior when presented with multiple-choice questions followed by deliberate challenges to their fictitious previous response. The indices are specifically designed to detect and measure what could be characterized as sycophantic behavior (excessive agreement with user challenges) or stubborn responses (rigid adherence to the fictitious response in the chat history) from LLMs. These metrics allow investigation of the relationships between sycophancy, stubbornness, and the model's actual mastery of the subject matter. We demonstrate the utility of these indices using two physics problems as test scenarios with various OpenAI models. The framework is intentionally generalizable to any multiple-choice format question, including on topics without universally accepted correct answers. Our results reveal measurable differences across OpenAI model generations, with trends indicating that newer models and those employing greater "Reasoning Effort" exhibit reduced sycophantic behavior. The FR pairing method combined with our proposed indices provides a practical, adaptable toolkit for systematically comparing LLM dialogue behaviors across different models and contexts.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 46 references that are in the map. First posted on arXiv 2 Jan 2026.
Builds on
14 of 46 references.
References · 14
- 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
- 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
- 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
- 2025SycEval: Evaluating LLM Sycophancycited by 108
- 2024Sycophancy in Large Language Models: Causes and Mitigationscited by 101
- 2021Socially Responsible AI Algorithms: Issues, Purposes, and Challengescited by 56
- 2025When Truth Is Overridden: Uncovering the Internal Origins of Sycophancy in Large Language Modelscited by 50
- 2024Do These LLM Benchmarks Agree? Fixing Benchmark Evaluation with BenchBenchcited by 27
- 2025BrokenMath: A Benchmark for Sycophancy in Theorem Proving with LLMscited by 27
- 2025Sycophancy under Pressure: Evaluating and Mitigating Sycophantic Bias via Adversarial Dialogues in Scientific QAcited by 18
- 2023Advances in apparent conceptual physics reasoning in GPT-4cited by 12
- 2025ReliableMath: Benchmark of Reliable Mathematical Reasoning on Large Language Modelscited by 8
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Beyond the Sycophancy Score: How Task, Model, and Pressure Shape LLM Yielding
- 2024Self-Preference Bias in LLM-as-a-Judge
- 2025Is LLM an Overconfident Judge? Unveiling the Capabilities of LLMs in Detecting Offensive Language with Annotation Disagreement
- 2024FB-Bench: A Fine-Grained Multi-Task Benchmark for Evaluating LLMs' Responsiveness to Human Feedback
- 2026When Consistency Does Not Mean Reliability: Evaluating Local LLM Judges Against Human Ratings
- 2024Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Models
- 2026What Does Multi-Agent LLM Debate Actually Change? A Layered Analysis of Disagreement and Answer Quality
- 2026SPINE: Measuring LLM Sycophancy under Sustained Multi-Turn Pressure