mlsanityData to 7 Oct 2026

Break It Down: A Question Understanding Benchmark

arXiv 2001.11770 ↗First posted 31 Jan 2020cs.CLcited by 131

Not on the map. Why?

Cited by 22 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Understanding natural language questions entails the ability to break down a question into the requisite steps for computing its answer. In this work, we introduce a Question Decomposition Meaning Representation (QDMR) for questions. QDMR constitutes the ordered list of steps, expressed through natural language, that are necessary for answering a question. We develop a crowdsourcing pipeline, showing that quality QDMRs can be annotated at scale, and release the Break dataset, containing over 83K pairs of questions and their QDMRs. We demonstrate the utility of QDMR by showing that (a) it can be used to improve open-domain question answering on the HotpotQA dataset, (b) it can be deterministically converted to a pseudo-SQL formal language, which can alleviate annotation in semantic parsing applications. Last, we use Break to train a sequence-to-sequence model with copying that parses questions into QDMR structures, and show that it substantially outperforms several natural baselines.

Key facts. Cited by 131 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 Jan 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 25; 2022: 29; 2023: 34; 2024: 13; 2025: 18; 2026: 12.

131 papers.

Sort:

By area:

  1. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  2. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  3. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  4. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  5. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  6. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 364
  7. 2022Binding Language Models in Symbolic Languagescited by 224
  8. 2023An LLM Compiler for Parallel Function Callingcited by 170
  9. 2021Constrained Language Models Yield Few-Shot Semantic Parserscited by 162
  10. 2022MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoningcited by 153
  11. 2023Compositional Exemplars for In-context Learningcited by 143
  12. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
38 more
  1. 2023Unified Demonstration Retriever for In-Context Learningcited by 141
  2. 2023Complex Claim Verification with Evidence Retrieved in the Wildcited by 110
  3. 2021Case-based Reasoning for Natural Language Queries over Knowledge Basescited by 107
  4. 2023Answering Questions by Meta-Reasoning over Multiple Chains of Thoughtcited by 87
  5. 2023Exploring Chain-of-Thought Style Prompting for Text-to-SQLcited by 69
  6. 2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 69
  7. 2023Natural Language Reasoning, A Surveycited by 61
  8. 2023Coverage-based Example Selection for In-Context Learningcited by 58
  9. 2022GraphWorld: Fake Graphs Bring Real Insights for GNNscited by 56
  10. 2024A Survey of Text-to-SQL in the Era of LLMs: Where are we, and where are we going?cited by 54
  11. 2024What Are Tools Anyway? A Survey from the Language Model Perspectivecited by 53
  12. 2022Reasoning Like Program Executorscited by 45
  13. 2022Is a Question Decomposition Unit All We Need?cited by 37
  14. 2021Exploring Universal Intrinsic Task Subspace via Prompt Tuningcited by 25
  15. 2023CRUSH4SQL: Collective Retrieval Using Schema Hallucination For Text2SQLcited by 23
  16. 2022Reasoning over Public and Private Data in Retrieval-Based Systemscited by 21
  17. 2021Factoring Statutory Reasoning as Language Understanding Challengescited by 20
  18. 2021Database Reasoning Over Textcited by 20
  19. 2022Learning to Decompose: Hypothetical Question Decomposition Based on Comparable Textscited by 18
  20. 2024Chain of Logic: Rule-Based Reasoning with Large Language Modelscited by 18
  21. 2024BIRD: A Trustworthy Bayesian Inference Framework for Large Language Modelscited by 18
  22. 2024Divide-or-Conquer? Which Part Should You Distill Your LLM?cited by 17
  23. 2024Metasql: A Generate-then-Rank Framework for Natural Language to SQL Translationcited by 17
  24. 2021Weakly Supervised Text-to-SQL Parsing through Question Decompositioncited by 15
  25. 2022Measuring Compositional Consistency for Video Question Answeringcited by 15
  26. 2023Question Decomposition Tree for Answering Complex Questions over Knowledge Basescited by 15
  27. 2025EnrichIndex: Using LLMs to Enrich Retrieval Indices Offlinecited by 15
  28. 2021NOAHQA: Numerical Reasoning with Interpretable Graph Question Answering Datasetcited by 14
  29. 2025TempRetriever: Fusion-based Temporal Dense Passage Retrieval for Time-Sensitive Questionscited by 14
  30. 2021Hey AI, Can You Solve Complex Tasks by Talking to Agents?cited by 13
  31. 2022GraphQ IR: Unifying the Semantic Parsing of Graph Query Languages with One Intermediate Representationcited by 13
  32. 2023Querying Large Language Models with SQLcited by 13
  33. 2023Prompting with Pseudo-Code Instructionscited by 13
  34. 2023EchoPrompt: Instructing the Model to Rephrase Queries for Improved In-context Learningcited by 13
  35. 2023Semantic Decomposition of Question and SQL for Text-to-SQL Parsingcited by 13
  36. 2022Eliciting and Understanding Cross-Task Skills with Task-Level Mixture-of-Expertscited by 12
  37. 2023Generating Data for Symbolic Language with Large Language Modelscited by 12
  38. 2024Analyzing the Role of Semantic Representations in the Era of Large Language Modelscited by 12

50 most cited of 131 All 131 (JSON)