mlsanityData to 7 Oct 2026

DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation

arXiv 2211.11501 ↗First posted 18 Nov 2022cs.SE cs.CLcited by 442

Not on the map. Why?

Cited by 115 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce DS-1000, a code generation benchmark with a thousand data science problems spanning seven Python libraries, such as NumPy and Pandas. Compared to prior works, DS-1000 incorporates three core features. First, our problems reflect diverse, realistic, and practical use cases since we collected them from StackOverflow. Second, our automatic evaluation is highly specific (reliable) -- across all Codex-002-predicted solutions that our evaluation accept, only 1.8% of them are incorrect; we achieve this with multi-criteria metrics, checking both functional correctness by running test cases and surface-form constraints by restricting API usages or keywords. Finally, we proactively defend against memorization by slightly modifying our problems to be different from the original StackOverflow source; consequently, models cannot answer them correctly by memorizing the solutions from pre-training. The current best public system (Codex-002) achieves 43.3% accuracy, leaving ample room for improvement. We release our benchmark at https://ds1000-code-gen.github.io.

Key facts. Cited by 442 papers in the map (data through 7 Oct 2026). Builds on 23 of its 38 references that are in the map. First posted on arXiv 18 Nov 2022.

Builds on

23 of 38 references.

References · 23

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2021Program Synthesis with Large Language Modelscited by 3,509
  3. 2020Extracting Training Data from Large Language Modelscited by 2,144
  4. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  5. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  6. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  7. 2021Measuring Coding Challenge Competence With APPScited by 879
  8. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  9. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  10. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  11. 2021PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Modelscited by 402
  12. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
11 more
  1. 2022Efficient Training of Language Models to Fill in the Middlecited by 248
  2. 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
  3. 2022Synchromesh: Reliable code generation from pre-trained language modelscited by 156
  4. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  5. 2020Semantic Evaluation for Text-to-SQL with Distilled Test Suitescited by 123
  6. 2022Natural Language to Code Translation with Executioncited by 122
  7. 2020Unit Test Case Generation with Transformers and Focal Contextcited by 118
  8. 2021Memorization vs. Generalization: Quantifying Data Leakage in NLP Performance Evaluationcited by 69
  9. 2019JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generationcited by 49
  10. 2022Training and Evaluating a Jupyter Notebook Data Science Assistantcited by 31
  11. 2022HELoC: Hierarchical Contrastive Learning of Source Code Representationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 55; 2024: 142; 2025: 118; 2026: 121.

442 papers.

Sort:

By area:

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  2. 2023A Survey of Large Language Modelscited by 2,537
  3. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  4. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  5. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  6. 2023StarCoder: may the source be with you!cited by 790
  7. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  8. 2024A Survey on Large Language Models for Code Generationcited by 646
  9. 2023A Comprehensive Overview of Large Language Modelscited by 639
  10. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  11. 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 400
  12. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
38 more
  1. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  2. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  3. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 251
  4. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 229
  5. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 228
  6. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
  7. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 223
  8. 2025KernelBench: Can LLMs Write Efficient GPU Kernels?cited by 206
  9. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 196
  10. 2025Survey on Evaluation of LLM-based Agentscited by 190
  11. 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasetscited by 188
  12. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  13. 2024SciCode: A Research Coding Benchmark Curated by Scientistscited by 172
  14. 2023SantaCoder: don't reach for the stars!cited by 169
  15. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 166
  16. 2022Large Language Models Meet NL2Code: A Surveycited by 135
  17. 2024InfiAgent-DABench: Evaluating Agents on Data Analysis Taskscited by 134
  18. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 129
  19. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  20. 2024OpenCoder: The Open Cookbook for Top-Tier Code Large Language Modelscited by 127
  21. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 124
  22. 2024DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoningcited by 119
  23. 2024ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generationcited by 118
  24. 2023Who Wrote this Code? Watermarking for Code Generationcited by 116
  25. 2024Granite Code Models: A Family of Open Foundation Models for Code Intelligencecited by 116
  26. 2024Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Codecited by 106
  27. 2023CodeBERTScore: Evaluating Code Generation with Pretrained Models of Codecited by 101
  28. 2023Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakescited by 101
  29. 2024SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulationcited by 100
  30. 2024MatPlotAgent: Method and Evaluation for LLM-Based Agentic Scientific Data Visualizationcited by 98
  31. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoningcited by 96
  32. 2024LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigationcited by 92
  33. 2022Execution-Based Evaluation for Open-Domain Code Generationcited by 89
  34. 2024CodeRAG-Bench: Can Retrieval Augment Code Generation?cited by 89
  35. 2024Tuning Language Models by Proxycited by 88
  36. 2022Natural Language to Code Generation in Interactive Data Science Notebookscited by 86
  37. 2025From Automation to Autonomy: A Survey on Large Language Models in Scientific Discoverycited by 86
  38. 2023TaskWeaver: A Code-First Agent Frameworkcited by 85

50 most cited of 442 All 442 (JSON)