# Quick and (not so) Dirty: Unsupervised Selection of Justification Sentences for Multi-hop Question Answering

arXiv: https://arxiv.org/abs/1911.07176 · HTML: https://mlsanity.com/paper/1911.07176/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Nov 2019.

## Built on by (most cited first, 30 of 59)

- [ORPO: Monolithic Preference Optimization without Reference Model](https://mlsanity.com/paper/2403.07691/) (2024)
- [ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers](https://mlsanity.com/paper/2206.01861/) (2022)
- [Who's Harry Potter? Approximate Unlearning in LLMs](https://mlsanity.com/paper/2310.02238/) (2023)
- [Better & Faster Large Language Models via Multi-token Prediction](https://mlsanity.com/paper/2404.19737/) (2024)
- [The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits](https://mlsanity.com/paper/2402.17764/) (2024)
- [Guardrail Baselines for Unlearning in LLMs](https://mlsanity.com/paper/2403.03329/) (2024)
- [MARS: Unleashing the Power of Variance Reduction for Training Large Models](https://mlsanity.com/paper/2411.10438/) (2024)
- [Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2112.08688/) (2021)
- [BitNet b1.58 2B4T Technical Report](https://mlsanity.com/paper/2504.12285/) (2025)
- [Tensor Product Attention Is All You Need](https://mlsanity.com/paper/2501.06425/) (2025)
- [Scalable MatMul-free Language Modeling](https://mlsanity.com/paper/2406.02528/) (2024)
- [Q-Sparse: All Large Language Models can be Fully Sparsely-Activated](https://mlsanity.com/paper/2407.10969/) (2024)
- [JustLogic: A Comprehensive Benchmark for Evaluating Deductive Reasoning in Large Language Models](https://mlsanity.com/paper/2501.14851/) (2025)
- [RECOST: External Knowledge Guided Data-efficient Instruction Tuning](https://mlsanity.com/paper/2402.17355/) (2024)
- [BitNet v2: Native 4-bit Activations with Hadamard Transformation for 1-bit LLMs](https://mlsanity.com/paper/2504.18415/) (2025)
- [Dynamic Semantic Graph Construction and Reasoning for Explainable Multi-hop Science Question Answering](https://mlsanity.com/paper/2105.11776/) (2021)
- [METGEN: A Module-Based Entailment Tree Generation Framework for Answer Explanation](https://mlsanity.com/paper/2205.02593/) (2022)
- [NELLIE: A Neuro-Symbolic Inference Engine for Grounded, Compositional, and Explainable Reasoning](https://mlsanity.com/paper/2209.07662/) (2022)
- [Exploiting Reasoning Chains for Multi-hop Science Question Answering](https://mlsanity.com/paper/2109.02905/) (2021)
- [Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptation](https://mlsanity.com/paper/2405.17484/) (2024)
- [DeepSpeed Data Efficiency: Improving Deep Learning Model Quality and Training Efficiency via Efficient Data Sampling and Routing](https://mlsanity.com/paper/2212.03597/) (2022)
- [InfiGFusion: Graph-on-Logits Distillation via Efficient Gromov-Wasserstein for Model Fusion](https://mlsanity.com/paper/2505.13893/) (2025)
- [Random-LTD: Random and Layerwise Token Dropping Brings Efficient Training for Large-scale Transformers](https://mlsanity.com/paper/2211.11586/) (2022)
- [Exploring Quantization for Efficient Pre-Training of Transformer Language Models](https://mlsanity.com/paper/2407.11722/) (2024)
- [Single-pass Detection of Jailbreaking Input in Large Language Models](https://mlsanity.com/paper/2502.15435/) (2025)
- [Hardware-Efficient Attention for Fast Decoding](https://mlsanity.com/paper/2505.21487/) (2025)
- [MARS-M: When Variance Reduction Meets Matrices](https://mlsanity.com/paper/2510.21800/) (2025)
- [Deep Delta Learning](https://mlsanity.com/paper/2601.00417/) (2026)
- [InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Models](https://mlsanity.com/paper/2505.13878/) (2025)
- [GradPower: Powering Gradients for Faster Language Model Pre-Training](https://mlsanity.com/paper/2505.24275/) (2025)

## Abstract (arXiv)

We propose an unsupervised strategy for the selection of justification sentences for multi-hop question answering (QA) that (a) maximizes the relevance of the selected sentences, (b) minimizes the overlap between the selected facts, and (c) maximizes the coverage of both question and answer. This unsupervised sentence selection method can be coupled with any supervised QA approach. We show that the sentences selected by our method improve the performance of a state-of-the-art supervised QA model on two multi-hop QA datasets: AI2's Reasoning Challenge (ARC) and Multi-Sentence Reading Comprehension (MultiRC). We obtain new state-of-the-art performance on both datasets among approaches that do not use external resources for training the QA system: 56.82% F1 on ARC (41.24% on Challenge and 64.49% on Easy) and 26.1% EM0 on MultiRC. Our justification sentences have higher quality than the justifications selected by a strong information retrieval baseline, e.g., by 5.4% F1 in MultiRC. We also show that our unsupervised selection of justification sentences is more stable across domains than a state-of-the-art supervised sentence selection method.