# Break It Down: A Question Understanding Benchmark

arXiv: https://arxiv.org/abs/2001.11770 · HTML: https://mlsanity.com/paper/2001.11770/

## Key facts

- Cited by 131 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Jan 2020.

## Built on by (most cited first, 30 of 131)

- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [MuSiQue: Multihop Questions via Single-hop Question Composition](https://mlsanity.com/paper/2108.00573/) (2021)
- [Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies](https://mlsanity.com/paper/2101.02235/) (2021)
- [Learning To Retrieve Prompts for In-Context Learning](https://mlsanity.com/paper/2112.08633/) (2021)
- [Decomposed Prompting: A Modular Approach for Solving Complex Tasks](https://mlsanity.com/paper/2210.02406/) (2022)
- [Making Retrieval-Augmented Language Models Robust to Irrelevant Context](https://mlsanity.com/paper/2310.01558/) (2023)
- [Binding Language Models in Symbolic Languages](https://mlsanity.com/paper/2210.02875/) (2022)
- [An LLM Compiler for Parallel Function Calling](https://mlsanity.com/paper/2312.04511/) (2023)
- [Constrained Language Models Yield Few-Shot Semantic Parsers](https://mlsanity.com/paper/2104.08768/) (2021)
- [MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning](https://mlsanity.com/paper/2205.00445/) (2022)
- [Compositional Exemplars for In-context Learning](https://mlsanity.com/paper/2302.05698/) (2023)
- [CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP](https://mlsanity.com/paper/2104.08835/) (2021)
- [Unified Demonstration Retriever for In-Context Learning](https://mlsanity.com/paper/2305.04320/) (2023)
- [Complex Claim Verification with Evidence Retrieved in the Wild](https://mlsanity.com/paper/2305.11859/) (2023)
- [Case-based Reasoning for Natural Language Queries over Knowledge Bases](https://mlsanity.com/paper/2104.08762/) (2021)
- [Answering Questions by Meta-Reasoning over Multiple Chains of Thought](https://mlsanity.com/paper/2304.13007/) (2023)
- [In-context Learning with Retrieved Demonstrations for Language Models: A Survey](https://mlsanity.com/paper/2401.11624/) (2024)
- [Exploring Chain-of-Thought Style Prompting for Text-to-SQL](https://mlsanity.com/paper/2305.14215/) (2023)
- [Natural Language Reasoning, A Survey](https://mlsanity.com/paper/2303.14725/) (2023)
- [Coverage-based Example Selection for In-Context Learning](https://mlsanity.com/paper/2305.14907/) (2023)
- [GraphWorld: Fake Graphs Bring Real Insights for GNNs](https://mlsanity.com/paper/2203.00112/) (2022)
- [A Survey of Text-to-SQL in the Era of LLMs: Where are we, and where are we going?](https://mlsanity.com/paper/2408.05109/) (2024)
- [What Are Tools Anyway? A Survey from the Language Model Perspective](https://mlsanity.com/paper/2403.15452/) (2024)
- [Reasoning Like Program Executors](https://mlsanity.com/paper/2201.11473/) (2022)
- [Is a Question Decomposition Unit All We Need?](https://mlsanity.com/paper/2205.12538/) (2022)
- [Exploring Universal Intrinsic Task Subspace via Prompt Tuning](https://mlsanity.com/paper/2110.07867/) (2021)
- [CRUSH4SQL: Collective Retrieval Using Schema Hallucination For Text2SQL](https://mlsanity.com/paper/2311.01173/) (2023)
- [Reasoning over Public and Private Data in Retrieval-Based Systems](https://mlsanity.com/paper/2203.11027/) (2022)
- [Factoring Statutory Reasoning as Language Understanding Challenges](https://mlsanity.com/paper/2105.07903/) (2021)
- [Database Reasoning Over Text](https://mlsanity.com/paper/2106.01074/) (2021)

## Abstract (arXiv)

Understanding natural language questions entails the ability to break down a question into the requisite steps for computing its answer. In this work, we introduce a Question Decomposition Meaning Representation (QDMR) for questions. QDMR constitutes the ordered list of steps, expressed through natural language, that are necessary for answering a question. We develop a crowdsourcing pipeline, showing that quality QDMRs can be annotated at scale, and release the Break dataset, containing over 83K pairs of questions and their QDMRs. We demonstrate the utility of QDMR by showing that (a) it can be used to improve open-domain question answering on the HotpotQA dataset, (b) it can be deterministically converted to a pseudo-SQL formal language, which can alleviate annotation in semantic parsing applications. Last, we use Break to train a sequence-to-sequence model with copying that parses questions into QDMR structures, and show that it substantially outperforms several natural baselines.