mlsanityData to 7 Oct 2026

Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI2 Reasoning Challenge

arXiv 2102.03315 ↗First posted 5 Feb 2021cs.CL cs.AIcited by 79

Not on the map. Why?

Abstract (arXiv)

We present the ARC-DA dataset, a direct-answer ("open response", "freeform") version of the ARC (AI2 Reasoning Challenge) multiple-choice dataset. While ARC has been influential in the community, its multiple-choice format is unrepresentative of real-world questions, and multiple choice formats can be particularly susceptible to artifacts. The ARC-DA dataset addresses these concerns by converting questions to direct-answer format using a combination of crowdsourcing and expert review. The resulting dataset contains 2985 questions with a total of 8436 valid answers (questions typically have more than one valid answer). ARC-DA is one of the first DA datasets of natural questions that often require reasoning, and where appropriate question decompositions are not evident from the questions themselves. We describe the conversion approach taken, appropriate evaluation metrics, and several strong models. Although high, the best scores (81% GENIE, 61.4% F1, 63.2% ROUGE-L) still leave considerable room for improvement. In addition, the dataset provides a natural setting for new research on explanation, as many questions require reasoning to construct answers. We hope the dataset spurs further advances in complex question-answering by the community. ARC-DA is available at https://allenai.org/data/arc-da

Key facts. Cited by 79 papers in the map (data through 7 Oct 2026). Builds on 20 of its 27 references that are in the map. First posted on arXiv 5 Feb 2021.

Builds on

20 of 27 references.

References · 20

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  6. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  7. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  8. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  9. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  10. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  11. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  12. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
8 more
  1. 2019QASC: A Dataset for Question Answering via Sentence Compositioncited by 248
  2. 2019Reasoning Over Paragraph Effects in Situationscited by 79
  3. 2018Modeling Naive Psychology of Characters in Simple Commonsense Storiescited by 59
  4. 2020Temporal Reasoning on Implicit Events from Distant Supervisioncited by 58
  5. 2018A Systematic Classification of Knowledge, Reasoning, and Context within the ARC Datasetcited by 54
  6. 2020Differentiable Open-Ended Commonsense Reasoningcited by 29
  7. 2018Learning to Attend On Essential Terms: An Enhanced Retriever-Reader Model for Open-domain Question Answeringcited by 12
  8. 2018Answering Science Exam Questions Using Query Rewriting with Background Knowledgecited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 1; 2023: 10; 2024: 29; 2025: 19; 2026: 14.

79 papers.

Sort:

By area:

  1. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  2. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  3. 2024Corrective Retrieval Augmented Generationcited by 187
  4. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  5. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 155
  6. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  7. 2024Long Context Compression with Activation Beaconcited by 101
  8. 2023UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluationcited by 74
  9. 2024Harder Tasks Need More Experts: Dynamic Routing in MoE Modelscited by 71
  10. 2024Can multiple-choice questions really be useful in detecting the abilities of LLMs?cited by 66
  11. 2023A Multi-Task Embedder For Retrieval Augmented LLMscited by 63
  12. 2023Learning to Retrieve In-Context Examples for Large Language Modelscited by 60
38 more
  1. 2025Distillation Scaling Lawscited by 59
  2. 2021GooAQ: Open Question Answering with Diverse Answer Typescited by 53
  3. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 51
  4. 2023Adapting Large Language Models to Domains via Reading Comprehensioncited by 48
  5. 2023Won't Get Fooled Again: Answering Questions with False Premisescited by 47
  6. 2024The AdEMAMix Optimizer: Better, Faster, Oldercited by 47
  7. 2025Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Modelscited by 44
  8. 2021General-Purpose Question-Answering with Macawcited by 40
  9. 2024Optima: Optimizing Effectiveness and Efficiency for LLM-Based Multi-Agent Systemcited by 40
  10. 2024Prompting Techniques for Secure Code Generation: A Systematic Investigationcited by 33
  11. 2025Capability Instruction Tuning: A New Paradigm for Dynamic LLM Routingcited by 25
  12. 2021Zero-shot and Few-shot Learning with Knowledge Graphs: A Comprehensive Surveycited by 23
  13. 2024COMPL-AI Framework: A Technical Interpretation and LLM Benchmarking Suite for the EU Artificial Intelligence Actcited by 23
  14. 2024K-Level Reasoning: Establishing Higher Order Beliefs in Large Language Models for Strategic Reasoningcited by 21
  15. 2024MindMerger: Efficient Boosting LLM Reasoning in non-English Languagescited by 21
  16. 2025WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-trainingcited by 21
  17. 2024Agentic LLMs in the Supply Chain: Towards Autonomous Multi-Agent Consensus-Seekingcited by 16
  18. 2024SciEx: Benchmarking Large Language Models on Scientific Exams with Human Expert Grading and Automatic Gradingcited by 14
  19. 2024Delving into the Reversal Curse: How Far Can Large Language Models Generalize?cited by 11
  20. 2025How Much Knowledge Can You Pack into a LoRA Adapter without Harming LLM?cited by 11
  21. 2024Does Combining Parameter-efficient Modules Improve Few-shot Transfer Accuracy?cited by 9
  22. 2024MaskMoE: Boosting Token-Level Learning via Routing Mask in Mixture-of-Expertscited by 9
  23. 2025Self-Routing RAG: Binding Selective Retrieval with Knowledge Verbalizationcited by 9
  24. 2022ELQA: A Corpus of Metalinguistic Questions and Answers about Englishcited by 8
  25. 2024Se2: Sequential Example Selection for In-Context Learningcited by 8
  26. 2023NuclearQA: A Human-Made Benchmark for Language Models for the Nuclear Domaincited by 7
  27. 2025Exploiting Vocabulary Frequency Imbalance in Language Model Pre-trainingcited by 7
  28. 2026MergeMix: Optimizing Mid-Training Data Mixtures via Learnable Model Mergingcited by 7
  29. 2024DropBP: Accelerating Fine-Tuning of Large Language Models by Dropping Backward Propagationcited by 6
  30. 2024Mix-CPT: A Domain Adaptation Framework via Decoupling Knowledge Learning and Format Alignmentcited by 6
  31. 2024Naive Bayes-based Context Extension for Large Language Modelscited by 5
  32. 2024Enhancing Language Model Rationality with Bi-Directional Deliberation Reasoningcited by 5
  33. 2024Cross-layer Attention Sharing for Pre-trained Large Language Modelscited by 5
  34. 2024MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Modelscited by 5
  35. 2025Entropy-Guided Token Dropout: Training Autoregressive Language Models with Limited Domain Datacited by 5
  36. 2026ORCH: many analyses, one merge-a deterministic multi-agent orchestrator for discrete-choice reasoning with EMA-guided routingcited by 5
  37. 2021Enriching a Model's Notion of Belief using a Persistent Memorycited by 4
  38. 2023Can an AI Win Ghana's National Science and Maths Quiz? An AI Grand Challenge for Educationcited by 4

50 most cited of 79 All 79 (JSON)