mlsanityData to 7 Oct 2026

Crowdsourcing Multiple Choice Science Questions

arXiv 1707.06209 ↗First posted 19 Jul 2017cs.HC cs.AI cs.CLcited by 742

Not on the map. Why?

Cited by 54 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present a novel method for obtaining high-quality, domain-targeted multiple choice questions from crowd workers. Generating these questions can be difficult without trading away originality, relevance or diversity in the answer options. Our method addresses these problems by leveraging a large corpus of domain-specific text and a small set of existing questions. It produces model suggestions for document selection and answer distractor choice which aid the human question generation process. With this method we have assembled SciQ, a dataset of 13.7K multiple choice science exam questions (Dataset available at http://allenai.org/data.html). We demonstrate that the method produces in-domain questions by providing an analysis of this new dataset and by showing that humans cannot distinguish the crowdsourced questions from original questions. When using SciQ as additional training data to existing questions, we observe accuracy improvements on real science exams.

Key facts. Cited by 742 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Jul 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 25; 2023: 55; 2024: 187; 2025: 254; 2026: 211.

742 papers.

Sort:

By area:

  1. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  2. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  3. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  4. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  5. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  6. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  7. 2023A Comprehensive Overview of Large Language Modelscited by 639
  8. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  9. 2024Scaling and evaluating sparse autoencoderscited by 530
  10. 2021MetaICL: Learning to Learn In Contextcited by 450
  11. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  12. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
38 more
  1. 2023Scaling Data-Constrained Language Modelscited by 391
  2. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 300
  3. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  4. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  5. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 238
  6. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  7. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  8. 2023Power Hungry Processing: Watts Driving the Cost of AI Deployment?cited by 186
  9. 2024Large Language Models for Education: A Survey and Outlookcited by 185
  10. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 178
  11. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 172
  12. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 156
  13. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 155
  14. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  15. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 149
  16. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
  17. 2023Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Modelscited by 147
  18. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  19. 2025MMVU: Measuring Expert-Level Multi-Discipline Video Understandingcited by 131
  20. 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124
  21. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 122
  22. 2025Are Sparse Autoencoders Useful? A Case Study in Sparse Probingcited by 109
  23. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 105
  24. 2025Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoningcited by 101
  25. 2024Large Language Models Must Be Taught to Know What They Don't Knowcited by 99
  26. 2024MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Modelscited by 96
  27. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  28. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 96
  29. 2024Understanding Emergent Abilities of Language Models from the Loss Perspectivecited by 94
  30. 2022What Language Model to Train if You Have One Million GPU Hours?cited by 92
  31. 2024Med42-v2: A Suite of Clinical LLMscited by 89
  32. 2023DoGE: Domain Reweighting with Generalization Estimationcited by 87
  33. 2024Tokenization Is More Than Compressioncited by 87
  34. 2024On Verbalized Confidence Scores for LLMscited by 83
  35. 2023How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questionscited by 82
  36. 2024Hyper-Connectionscited by 81
  37. 2023Exploring the Benefits of Training Expert Language Models over Instruction Tuningcited by 80
  38. 2024Scientific Large Language Models: A Survey on Biological & Chemical Domainscited by 80

50 most cited of 742 All 742 (JSON)