# DSCodeBench: A Realistic Benchmark for Data Science Code Generation

arXiv: https://arxiv.org/abs/2505.15621 · HTML: https://mlsanity.com/paper/2505.15621/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 May 2025.

## Built on by (most cited first, 17 of 17)

- [DS-STAR: Data Science Agent for Solving Diverse Tasks across Heterogeneous Formats and Open-Ended Queries](https://mlsanity.com/paper/2509.21825/) (2025)
- [StatEval: A Comprehensive Benchmark for Large Language Models in Statistics](https://mlsanity.com/paper/2510.09517/) (2025)
- [LongDA: Benchmarking LLM Agents for Long-Document Data Analysis](https://mlsanity.com/paper/2601.02598/) (2026)
- [DSAEval: Evaluating Data Science Agents on a Wide Range of Real-World Data Science Problems](https://mlsanity.com/paper/2601.13591/) (2026)
- [JAMER: Project-Level Code Framework Dataset and Benchmark on Professional Game Engines](https://mlsanity.com/paper/2606.19830/) (2026)
- [Measuring Data Science Automation: A Survey of Evaluation Tools for AI Assistants and Agents](https://mlsanity.com/paper/2506.08800/) (2025)
- [CODA-BENCH: Can Code Agents Handle Data-Intensive Tasks?](https://mlsanity.com/paper/2606.15300/) (2026)
- [ContractEval: A Benchmark for Evaluating Contract-Satisfying Assertions in Code Generation](https://mlsanity.com/paper/2510.12047/) (2025)
- [Are LLMs Ready for Scientific Discovery? A Capability-Oriented Benchmark for AI Scientists](https://mlsanity.com/paper/2607.11079/) (2026)
- [To Add Is Machine, To Delete Is Human: Measuring and Mitigating Deletion Avoidance in LLM Code Editing](https://mlsanity.com/paper/2607.28887/) (2026)
- [ML2B: Benchmarking LLMs on Cross-Lingual ML Pipeline Generation](https://mlsanity.com/paper/2509.22768/) (2025)
- [From Mirage to Grounding: Towards Reliable Multimodal Circuit-to-Verilog Code Generation](https://mlsanity.com/paper/2604.27969/) (2026)
- [SciCodePile: A 128GB Corpus and Executable Benchmark for Challenging Scientific Code Generation](https://mlsanity.com/paper/2607.19104/) (2026)
- [VibeMemBench: Evaluating Memory Systems for Coding Agents on Real Repository Coding Tasks](https://mlsanity.com/paper/2609.23570/) (2026)
- [Visible Reasoning Is Not a Universal Optimizer: Persona- and Thinking-Dependent Effects in Analytics Code Generation](https://mlsanity.com/paper/2610.10639/) (2026)
- [HERO: Harnessing Evidence for Test-time Refinement and Prompt Optimization in Program Generation](https://mlsanity.com/paper/2610.19184/) (2026)
- [DUALVIEW: Leveraging Broad and Output-Specific Evidence for LLM-based Multi-Output Text Regression](https://mlsanity.com/paper/2610.19206/) (2026)

## Abstract (arXiv)

We introduce DSCodeBench, a new benchmark designed to evaluate large language models (LLMs) on complicated and realistic data science code generation tasks. DSCodeBench consists of 1,000 carefully constructed problems sourced from realistic problems from GitHub across ten widely used Python data science libraries. DSCodeBench offers a more challenging and representative testbed, more complex code solutions, more comprehensive data science libraries, clearer and better structured problem descriptions, and stronger test suites. To construct the DSCodeBench, we develop a robust pipeline that combines task scope selection, code construction, test case generation, and problem description synthesis. The process is paired with rigorous manual editing to ensure alignment and enhance the reliability of the evaluation. Experimental result shows that DSCodeBench exhibits robust scaling behavior, where larger models systematically outperform smaller ones, validating its ability to distinguish model capabilities. The best LLM we test, GPT-4o, has a pass@1 of 0.392, indicating that LLMs still have a large room to improve for realistic data science code generation tasks. We believe DSCodeBench will serve as a rigorous and trustworthy foundation for advancing LLM-based data science programming.