# Long Context Question Answering via Supervised Contrastive Learning

arXiv: https://arxiv.org/abs/2112.08777 · HTML: https://mlsanity.com/paper/2112.08777/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 19 references that are in the map.
- First posted on arXiv 16 Dec 2021.

## Builds on (most cited first, 13 of 13)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers](https://mlsanity.com/paper/2105.03011/) (2021)
- [Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning](https://mlsanity.com/paper/2011.01403/) (2020)
- [Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models](https://mlsanity.com/paper/2005.10389/) (2020)
- [Hierarchical Question Answering for Long Documents](https://mlsanity.com/paper/1611.01839/) (2016)

## Built on by (most cited first, 17 of 17)

- [PEARL: Prompting Large Language Models to Plan and Execute Actions Over Long Documents](https://mlsanity.com/paper/2305.14564/) (2023)
- [LongRAG: A Dual-Perspective Retrieval-Augmented Generation Paradigm for Long-Context Question Answering](https://mlsanity.com/paper/2410.18050/) (2024)
- [Peek Across: Improving Multi-Document Modeling via Cross-Document Question-Answering](https://mlsanity.com/paper/2305.15387/) (2023)
- [UncertaintyRAG: Span-Level Uncertainty Enhanced Long-Context Modeling for Retrieval-Augmented Generation](https://mlsanity.com/paper/2410.02719/) (2024)
- [MDCure: A Scalable Pipeline for Multi-Document Instruction-Following](https://mlsanity.com/paper/2410.23463/) (2024)
- [Reducing Distraction in Long-Context Language Models by Focused Learning](https://mlsanity.com/paper/2411.05928/) (2024)
- [SEAM: A Stochastic Benchmark for Multi-Document Tasks](https://mlsanity.com/paper/2406.16086/) (2024)
- [What Really Matters in Many-Shot Attacks? An Empirical Study of Long-Context Vulnerabilities in LLMs](https://mlsanity.com/paper/2505.19773/) (2025)
- [Document Structure in Long Document Transformers](https://mlsanity.com/paper/2401.17658/) (2024)
- [Medical Question Summarization with Entity-driven Contrastive Learning](https://mlsanity.com/paper/2304.07437/) (2023)
- [QontSum: On Contrasting Salient Content for Query-focused Summarization](https://mlsanity.com/paper/2307.07586/) (2023)
- [AttenWalker: Unsupervised Long-Document Question Answering via Attention-based Graph Walking](https://mlsanity.com/paper/2305.02235/) (2023)
- [Momentum Contrastive Pre-training for Question Answering](https://mlsanity.com/paper/2212.05762/) (2022)
- [DoubleDipper: Improving Long-Context LLMs via Context Recycling](https://mlsanity.com/paper/2406.13632/) (2024)
- [Long-context Non-factoid Question Answering in Indic Languages](https://mlsanity.com/paper/2504.13615/) (2025)
- [A Survey of the State-of-the-Art in Conversational Question Answering Systems](https://mlsanity.com/paper/2509.05716/) (2025)
- [TerraAgent: Demonstration-Grounded Multi-Agent Reasoning for Long-Context Remote Sensing Visual Question Answering](https://mlsanity.com/paper/2610.12517/) (2026)

## Abstract (arXiv)

Long-context question answering (QA) tasks require reasoning over a long document or multiple documents. Addressing these tasks often benefits from identifying a set of evidence spans (e.g., sentences), which provide supporting evidence for answering the question. In this work, we propose a novel method for equipping long-context QA models with an additional sequence-level objective for better identification of the supporting evidence. We achieve this via an additional contrastive supervision signal in finetuning, where the model is encouraged to explicitly discriminate supporting evidence sentences from negative ones by maximizing question-evidence similarity. The proposed additional loss exhibits consistent improvements on three different strong long-context transformer models, across two challenging question answering benchmarks -- HotpotQA and QAsper.