# It Is Not About What You Say, It Is About How You Say It: A Surprisingly Simple Approach for Improving Reading Comprehension

arXiv: https://arxiv.org/abs/2406.16779 · HTML: https://mlsanity.com/paper/2406.16779/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 68 references that are in the map.
- First posted on arXiv 24 Jun 2024.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023)
- [Do Prompt-Based Models Really Understand the Meaning of their Prompts?](https://mlsanity.com/paper/2109.01247/) (2021)
- [PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts](https://mlsanity.com/paper/2202.01279/) (2022)
- [Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4](https://mlsanity.com/paper/2304.03439/) (2023)
- [Large Language Models with Controllable Working Memory](https://mlsanity.com/paper/2211.05110/) (2022)
- [Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension](https://mlsanity.com/paper/2002.00293/) (2020)
- [Simple and Effective Multi-Paragraph Reading Comprehension](https://mlsanity.com/paper/1710.10723/) (2017)
- [Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering](https://mlsanity.com/paper/2306.04136/) (2023)
- [Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMs](https://mlsanity.com/paper/2311.02262/) (2023)
- [Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Models](https://mlsanity.com/paper/2208.07852/) (2022)
- [What do Models Learn from Question Answering Datasets?](https://mlsanity.com/paper/2004.03490/) (2020)
- [A Survey on Machine Reading Comprehension: Tasks, Evaluation Metrics and Benchmark Datasets](https://mlsanity.com/paper/2006.11880/) (2020)

## Built on by (most cited first, 4 of 4)

- [Adaptive Question Answering: Enhancing Language Model Proficiency for Addressing Knowledge Conflicts with Source Citations](https://mlsanity.com/paper/2410.04241/) (2024)
- [Asking Again and Again: Exploring LLM Robustness to Repeated Questions](https://mlsanity.com/paper/2412.07923/) (2024)
- [Lost in the Prompt Order: Revealing the Limitations of Causal Attention in Language Models](https://mlsanity.com/paper/2601.14152/) (2026)
- [MALAMUTE: A Multilingual, Highly-granular, Template-free, Education-based Probing Dataset](https://mlsanity.com/paper/2412.10105/) (2024)

## Abstract (arXiv)

Natural language processing has seen rapid progress over the past decade. Due to the speed of developments, some practices get established without proper evaluation. Considering one such case and focusing on reading comprehension, we ask our first research question: 1) How does the order of inputs -- i.e., question and context -- affect model performance? Additionally, given recent advancements in input emphasis, we ask a second research question: 2) Does emphasizing either the question, the context, or both enhance performance? Experimenting with 9 large language models across 3 datasets, we find that presenting the context before the question improves model performance, with an accuracy increase of up to 31%. Furthermore, emphasizing the context yields superior results compared to question emphasis, and in general, emphasizing parts of the input is particularly effective for addressing questions that models lack the parametric knowledge to answer. Experimenting with both prompt-based and attention-based emphasis methods, we additionally find that the best method is surprisingly simple: it only requires concatenating a few tokens to the input and results in an accuracy improvement of up to 36%, allowing smaller models to outperform their significantly larger counterparts.