# Single-Turn Debate Does Not Help Humans Answer Hard Reading-Comprehension Questions

arXiv: https://arxiv.org/abs/2204.05212 · HTML: https://mlsanity.com/paper/2204.05212/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 16 references that are in the map.
- First posted on arXiv 11 Apr 2022.

## Builds on (most cited first, 13 of 13)

- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [From Recognition to Cognition: Visual Commonsense Reasoning](https://mlsanity.com/paper/1811.10830/) (2018)
- [e-SNLI: Natural Language Inference with Natural Language Explanations](https://mlsanity.com/paper/1812.01193/) (2018)
- [AI safety via debate](https://mlsanity.com/paper/1805.00899/) (2018)
- [Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance](https://mlsanity.com/paper/2006.14779/) (2020)
- [Explain Yourself! Leveraging Language Models for Commonsense Reasoning](https://mlsanity.com/paper/1906.02361/) (2019)
- [Questioning the AI: Informing Design Practices for Explainable AI User Experiences](https://mlsanity.com/paper/2001.02478/) (2020)
- [QuALITY: Question Answering with Long Input Texts, Yes!](https://mlsanity.com/paper/2112.08608/) (2021)
- [On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection](https://mlsanity.com/paper/1811.07901/) (2018)
- [How do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-Interpretability of Explanation](https://mlsanity.com/paper/1802.00682/) (2018)
- [Quantifying Interpretability and Trust in Machine Learning Systems](https://mlsanity.com/paper/1901.08558/) (2019)
- [Joint Mind Modeling for Explanation Generation in Complex Human-Robot Collaborative Tasks](https://mlsanity.com/paper/2007.12803/) (2020)
- [Beneficial and Harmful Explanatory Machine Learning](https://mlsanity.com/paper/2009.06410/) (2020)

## Built on by (most cited first, 23 of 23)

- [Improving alignment of dialogue agents via targeted human judgements](https://mlsanity.com/paper/2209.14375/) (2022)
- [Self-critiquing models for assisting human evaluators](https://mlsanity.com/paper/2206.05802/) (2022)
- [The Alignment Problem from a Deep Learning Perspective](https://mlsanity.com/paper/2209.00626/) (2022)
- [Measuring Progress on Scalable Oversight for Large Language Models](https://mlsanity.com/paper/2211.03540/) (2022)
- [On scalable oversight with weak LLMs judging strong LLMs](https://mlsanity.com/paper/2407.04622/) (2024)
- [Large Language Models Help Humans Verify Truthfulness -- Except When They Are Convincingly Wrong](https://mlsanity.com/paper/2310.12558/) (2023)
- [In Search of Verifiability: Explanations Rarely Enable Complementary Performance in AI-Advised Decision Making](https://mlsanity.com/paper/2305.07722/) (2023)
- [How Susceptible are LLMs to Influence in Prompts?](https://mlsanity.com/paper/2408.11865/) (2024)
- [Two-Turn Debate Doesn't Help Humans Answer Hard Reading Comprehension Questions](https://mlsanity.com/paper/2210.10860/) (2022)
- [Training Language Models to Win Debates with Self-Play Improves Judge Accuracy](https://mlsanity.com/paper/2409.16636/) (2024)
- [An alignment safety case sketch based on debate](https://mlsanity.com/paper/2505.03989/) (2025)
- [Debate Helps Weak-to-Strong Generalization](https://mlsanity.com/paper/2501.13124/) (2025)
- [RedDebate: Safer Responses Through Multi-Agent Red Teaming Debates](https://mlsanity.com/paper/2506.11083/) (2025)
- [Playing Large Games with Oracles and AI Debate](https://mlsanity.com/paper/2312.04792/) (2023)
- [MODS: Moderating a Mixture of Document Speakers to Summarize Debatable Queries in Document Collections](https://mlsanity.com/paper/2502.00322/) (2025)
- [AI Debaters are More Persuasive when Arguing in Alignment with Their Own Beliefs](https://mlsanity.com/paper/2510.13912/) (2025)
- [Debate Training Reduces Reward Hacking in RLAIF](https://mlsanity.com/paper/2608.17776/) (2026)
- [Human-AI Complementarity: A Goal for Amplified Oversight](https://mlsanity.com/paper/2510.26518/) (2025)
- [Arguments that Alter Minds: LLM Rationales Sway Human (and LLM) Notions of Plausibility](https://mlsanity.com/paper/2510.08091/) (2025)
- [Reasoning Court: Combining Reasoning, Action, and Judgment for Multi-Hop Reasoning](https://mlsanity.com/paper/2504.09781/) (2025)
- [Debate Helps Weak Judges Reward Stronger Models](https://mlsanity.com/paper/2605.27483/) (2026)
- [When Helping Hurts and How to Fix It: Multi-Agent Debate for Data Cleaning](https://mlsanity.com/paper/2606.02866/) (2026)
- [Collaborative Disagreement Resolution for Scalable Oversight](https://mlsanity.com/paper/2607.01251/) (2026)

## Abstract (arXiv)

Current QA systems can generate reasonable-sounding yet false answers without explanation or evidence for the generated answer, which is especially problematic when humans cannot readily check the model's answers. This presents a challenge for building trust in machine learning systems. We take inspiration from real-world situations where difficult questions are answered by considering opposing sides (see Irving et al., 2018). For multiple-choice QA examples, we build a dataset of single arguments for both a correct and incorrect answer option in a debate-style set-up as an initial step in training models to produce explanations for two candidate answers. We use long contexts -- humans familiar with the context write convincing explanations for pre-selected correct and incorrect answers, and we test if those explanations allow humans who have not read the full context to more accurately determine the correct answer. We do not find that explanations in our set-up improve human accuracy, but a baseline condition shows that providing human-selected text snippets does improve accuracy. We use these findings to suggest ways of improving the debate set up for future data collection efforts.