mlsanityData to 7 Oct 2026

Training and Evaluating a Jupyter Notebook Data Science Assistant

arXiv 2201.12901 ↗First posted 30 Jan 2022cs.LG cs.SEcited by 31

Not on the map. Why?

Abstract (arXiv)

We study the feasibility of a Data Science assistant powered by a sequence-to-sequence transformer by training a new model JuPyT5 on all publicly available Jupyter Notebook GitHub repositories and developing a new metric: Data Science Problems (DSP). DSP is a collection of 1119 problems curated from 306 pedagogical notebooks with 92 dataset dependencies, natural language and Markdown problem descriptions, and assert-based unit tests. These notebooks were designed to test university students' mastery of various Python implementations of Math and Data Science, and we now leverage them to study the ability of JuPyT5 to understand and pass the tests. We analyze the content of DSP, validate its quality, and we find that given 100 sampling attempts JuPyT5 is able to solve 77.5% of the DSP problems. We further present various ablation and statistical analyses and compare DSP to other recent natural language to code benchmarks.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 18 of its 22 references that are in the map. First posted on arXiv 30 Jan 2022.

Builds on

18 of 22 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2021Program Synthesis with Large Language Modelscited by 3,509
  7. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  8. 2021Measuring Coding Challenge Competence With APPScited by 879
  9. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  10. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  11. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  12. 2020IntelliCode Compose: Code Generation Using Transformercited by 225
6 more
  1. 2019Gmail Smart Compose: Real-Time Assisted Writingcited by 97
  2. 2020PyMT5: multi-mode translation of natural language and Python code with transformerscited by 80
  3. 2019Pythia: AI-assisted Code Completion Systemcited by 54
  4. 2019JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generationcited by 49
  5. 2020Generating Accurate Assert Statements for Unit Test Cases using Pretrained Transformerscited by 30
  6. 2021Generating Bug-Fixes Using Pretrained Transformerscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 10; 2024: 9; 2025: 4; 2026: 2.

31 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Models for Code Generationcited by 646
  2. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  3. 2022Large Language Models Meet NL2Code: A Surveycited by 135
  4. 2022Execution-Based Evaluation for Open-Domain Code Generationcited by 89
  5. 2022Natural Language to Code Generation in Interactive Data Science Notebookscited by 86
  6. 2023PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedbackcited by 69
  7. 2025WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratchcited by 66
  8. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  9. 2024Agents in Software Engineering: Survey, Landscape, and Visioncited by 60
  10. 2023xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrievalcited by 46
  11. 2023SelfEvolve: A Code Evolution Framework via Large Language Modelscited by 43
  12. 2023CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generationcited by 39
19 more
  1. 2023SALLM: Security Assessment of Generated Codecited by 35
  2. 2022Fault-Aware Neural Code Rankerscited by 33
  3. 2022Execution-based Evaluation for Data Science Code Generation Modelscited by 31
  4. 2024InsightBench: Evaluating Business Analytics Agents Through Multi-Step Insight Generationcited by 30
  5. 2024MHPP: Exploring the Capabilities and Limitations of Language Models Beyond Basic Code Generationcited by 25
  6. 2023LLM for SoC Security: A Paradigm Shiftcited by 22
  7. 2024A Survey on Large Language Model-based Agents for Statistics and Data Sciencecited by 22
  8. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  9. 2024JavaBench: A Benchmark of Object-Oriented Code Generation for Evaluating Large Language Modelscited by 10
  10. 2024GitChameleon: Unmasking the Version-Switching Capabilities of Code Generation Modelscited by 6
  11. 2023Capture the Flag: Uncovering Data Insights with Large Language Modelscited by 4
  12. 2024CodeInsight: A Curated Dataset of Practical Coding Solutions from Stack Overflowcited by 4
  13. 2025Measuring Data Science Automation: A Survey of Evaluation Tools for AI Assistants and Agentscited by 4
  14. 2025Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Qualitycited by 3
  15. 2023Safurai 001: New Qualitative Approach for Code LLM Evaluationcited by 0
  16. 2023Safurai-Csharp: Harnessing Synthetic Data to improve language-specific Code LLMcited by 0
  17. 2024Linguacodus: A Synergistic Framework for Transformative Code Generation in Machine Learning Pipelinescited by 0
  18. 2026JupOtter: Cell-Level Bug Detection in Jupyter Notebookscited by 0
  19. 2026ORCA: Evaluating LLMs on Data Science Code Translationcited by 0