# Robustifying Multi-hop QA through Pseudo-Evidentiality Training

arXiv: https://arxiv.org/abs/2107.03242 · HTML: https://mlsanity.com/paper/2107.03242/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 23 references that are in the map.
- First posted on arXiv 7 Jul 2021.

## Builds on (most cited first, 21 of 21)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [Is Attention Interpretable?](https://mlsanity.com/paper/1906.03731/) (2019)
- [Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering](https://mlsanity.com/paper/1911.10470/) (2019)
- [An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models](https://mlsanity.com/paper/2007.06778/) (2020)
- [Don't Judge an Object by Its Context: Learning to Overcome Contextual Bias](https://mlsanity.com/paper/2001.03152/) (2020)
- [Compositional Questions Do Not Necessitate Multi-hop Reasoning](https://mlsanity.com/paper/1906.02900/) (2019)
- [Mind the Trade-off: Debiasing NLU Models without Degrading the In-distribution Performance](https://mlsanity.com/paper/2005.00315/) (2020)
- [Hierarchical Graph Network for Multi-hop Question Answering](https://mlsanity.com/paper/1911.03631/) (2019)
- [Dynamically Fused Graph Network for Multi-hop Reasoning](https://mlsanity.com/paper/1905.06933/) (2019)
- [Avoiding Reasoning Shortcuts: Adversarial Evaluation, Training, and Model Development for Multi-Hop QA](https://mlsanity.com/paper/1906.07132/) (2019)
- [Understanding Dataset Design Choices for Multi-hop Reasoning](https://mlsanity.com/paper/1904.12106/) (2019)
- [Revealing the Importance of Semantic Retrieval for Machine Reading at Scale](https://mlsanity.com/paper/1909.08041/) (2019)
- [Adversarial Regularization for Visual Question Answering: Strengths, Shortcomings, and Side Effects](https://mlsanity.com/paper/1906.08430/) (2019)
- [Finding Generalizable Evidence by Learning to Convince Q&A Models](https://mlsanity.com/paper/1909.05863/) (2019)
- [A Simple Yet Strong Pipeline for HotpotQA](https://mlsanity.com/paper/2004.06753/) (2020)
- [Unsupervised Alignment-based Iterative Evidence Retrieval for Multi-hop Question Answering](https://mlsanity.com/paper/2005.01218/) (2020)
- [A Self-Training Method for Machine Reading Comprehension with Soft Evidence Extraction](https://mlsanity.com/paper/2005.05189/) (2020)

## Built on by (most cited first, 13 of 13)

- [Natural Language Reasoning, A Survey](https://mlsanity.com/paper/2303.14725/) (2023)
- [Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2112.08688/) (2021)
- [Why LLMs Hallucinate, and How to Get (Evidential) Closure: Perceptual, Intensional, and Extensional Learning for Faithful Natural Language Generation](https://mlsanity.com/paper/2310.15355/) (2023)
- [Lost in the Middle, and In-Between: Enhancing Language Models' Ability to Reason Over Long Contexts in Multi-Hop QA](https://mlsanity.com/paper/2412.10079/) (2024)
- [Teaching Broad Reasoning Skills for Multi-Step QA by Generating Hard Contexts](https://mlsanity.com/paper/2205.12496/) (2022)
- [Understanding and Improving Zero-shot Multi-hop Reasoning in Generative Question Answering](https://mlsanity.com/paper/2210.04234/) (2022)
- [Counterfactual Multihop QA: A Cause-Effect Approach for Reducing Disconnected Reasoning](https://mlsanity.com/paper/2210.07138/) (2022)
- [Passage-Mask: A Learnable Regularization Strategy for Retriever-Reader Models](https://mlsanity.com/paper/2211.00915/) (2022)
- [Single Sequence Prediction over Reasoning Graphs for Multi-hop QA](https://mlsanity.com/paper/2307.00335/) (2023)
- [A Survey on Measuring and Mitigating Reasoning Shortcuts in Machine Reading Comprehension](https://mlsanity.com/paper/2209.01824/) (2022)
- [ECoRAG: Evidentiality-guided Compression for Long Context RAG](https://mlsanity.com/paper/2506.05167/) (2025)
- [Retrieval-Based Transformer for Table Augmentation](https://mlsanity.com/paper/2306.11843/) (2023)
- [Evidentiality-aware Retrieval for Overcoming Abstractiveness in Open-Domain Question Answering](https://mlsanity.com/paper/2304.03031/) (2023)

## Abstract (arXiv)

This paper studies the bias problem of multi-hop question answering models, of answering correctly without correct reasoning. One way to robustify these models is by supervising to not only answer right, but also with right reasoning chains. An existing direction is to annotate reasoning chains to train models, requiring expensive additional annotations. In contrast, we propose a new approach to learn evidentiality, deciding whether the answer prediction is supported by correct evidences, without such annotations. Instead, we compare counterfactual changes in answer confidence with and without evidence sentences, to generate "pseudo-evidentiality" annotations. We validate our proposed model on an original set and challenge set in HotpotQA, showing that our method is accurate and robust in multi-hop reasoning.