mlsanityData to 7 Oct 2026

Natural Language to Code Generation in Interactive Data Science Notebooks

arXiv 2212.09248 ↗First posted 19 Dec 2022cs.CL cs.SEcited by 86

Not on the map. Why?

Cited by 27 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Computational notebooks, such as Jupyter notebooks, are interactive computing environments that are ubiquitous among data scientists to perform data wrangling and analytic tasks. To measure the performance of AI pair programmers that automatically synthesize programs for those tasks given natural language (NL) intents from users, we build ARCADE, a benchmark of 1082 code generation problems using the pandas data analysis framework in data science notebooks. ARCADE features multiple rounds of NL-to-code problems from the same notebook. It requires a model to understand rich multi-modal contexts, such as existing notebook cells and their execution states as well as previous turns of interaction. To establish a strong baseline on this challenging task, we develop PaChiNCo, a 62B code language model (LM) for Python computational notebooks, which significantly outperforms public code LMs. Finally, we explore few-shot prompting strategies to elicit better code with step-by-step decomposition and NL explanation, showing the potential to improve the diversity and explainability of model predictions.

Key facts. Cited by 86 papers in the map (data through 7 Oct 2026). Builds on 54 of its 89 references that are in the map. First posted on arXiv 19 Dec 2022.

Builds on

54 of 89 references.

References · 54

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2021Program Synthesis with Large Language Modelscited by 3,509
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  10. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  11. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  12. 2021Measuring Coding Challenge Competence With APPScited by 879
38 more
  1. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  2. 2022PAL: Program-aided Language Modelscited by 635
  3. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  4. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  5. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  6. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  7. 2019AutoML: A Survey of the State-of-the-Artcited by 388
  8. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  9. 2020A Simple Language Model for Task-Oriented Dialoguecited by 331
  10. 2020Unsupervised Translation of Programming Languagescited by 262
  11. 2022UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Modelscited by 231
  12. 2022Binding Language Models in Symbolic Languagescited by 224
  13. 2022Memorizing Transformerscited by 219
  14. 2022Grounded Copilot: How Programmers Interact with Code-Generating Modelscited by 216
  15. 2019CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databasescited by 180
  16. 2019SParC: Cross-Domain Semantic Parsing in Contextcited by 141
  17. 2020Task-Oriented Dialogue as Dataflow Synthesiscited by 122
  18. 2020UBAR: Towards Fully End-to-End Task-Oriented Dialog Systems with GPT-2cited by 111
  19. 2021Towards Robustness of Text-to-SQL Models against Synonym Substitutioncited by 110
  20. 2021KaggleDBQA: Realistic Evaluation of Text-to-SQL Parserscited by 105
  21. 2021Jigsaw: Large Language Models meet Program Synthesiscited by 103
  22. 2022Evaluating the Text-to-SQL Capabilities of Large Language Modelscited by 101
  23. 2022Coder Reviewer Reranking for Code Generationcited by 83
  24. 2020NL4DV: A Toolkit for Generating Analytic Specifications for Data Visualization from Natural Language Queriescited by 73
  25. 2020AutoML to Date and Beyond: Challenges and Opportunitiescited by 63
  26. 2019JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generationcited by 49
  27. 2021SpreadsheetCoder: Formula Prediction from Semi-structured Contextcited by 41
  28. 2016Simpler Context-Dependent Logical Forms via Model Projectionscited by 33
  29. 2021AutoDS: Towards Human-Centered Automation of Data Sciencecited by 32
  30. 2022Training and Evaluating a Jupyter Notebook Data Science Assistantcited by 31
  31. 2022Execution-based Evaluation for Data Science Code Generation Modelscited by 31
  32. 2020TF-Coder: Program Synthesis for Tensor Manipulationscited by 27
  33. 2018Learning to Map Context-Dependent Sentences to Executable Formal Queriescited by 23
  34. 2021How Much Automation Does a Data Scientist Want?cited by 20
  35. 2021Documentation Matters: Human-Centered AI System to Assist Data Science Code Documentation in Computational Notebookscited by 17
  36. 2022Is MultiWOZ a Solved Task? An Interactive TOD Evaluation Framework with User Simulatorcited by 15
  37. 2015Larger-Context Language Modellingcited by 10
  38. 2020Data Engineering for Data Analytics: A Classification of the Issues, and Case Studiescited by 10

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 14; 2024: 27; 2025: 32; 2026: 12.

86 papers.

Sort:

By area:

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  2. 2023PaLM 2 Technical Reportcited by 1,154
  3. 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 400
  4. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  5. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 229
  6. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 223
  7. 2025KernelBench: Can LLMs Write Efficient GPU Kernels?cited by 206
  8. 2023Universal Self-Consistency for Large Language Model Generationcited by 198
  9. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 196
  10. 2024SciCode: A Research Coding Benchmark Curated by Scientistscited by 172
  11. 2025From Automation to Autonomy: A Survey on Large Language Models in Scientific Discoverycited by 86
  12. 2024BLADE: Benchmarking Language Model Agents for Data-Driven Sciencecited by 72
38 more
  1. 2022Parsel: Algorithmic Reasoning with Language Models by Composing Decompositionscited by 62
  2. 2023Clover: Closed-Loop Verifiable Code Generationcited by 62
  3. 2024EffiBench: Benchmarking the Efficiency of Automatically Generated Codecited by 62
  4. 2024NExT: Teaching Large Language Models to Reason about Code Executioncited by 60
  5. 2024DA-Code: Agent Data Science Code Generation Benchmark for Large Language Modelscited by 57
  6. 2024Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?cited by 55
  7. 2023SPAE: Semantic Pyramid AutoEncoder for Multimodal Generation with Frozen LLMscited by 54
  8. 2023Bias Testing and Mitigation in LLM-based Code Generationcited by 53
  9. 2023xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrievalcited by 46
  10. 2023LILO: Learning Interpretable Libraries by Compressing and Documenting Codecited by 46
  11. 2025DABstep: Data Agent Benchmark for Multi-step Reasoningcited by 44
  12. 2024Prompting Large Language Models to Tackle the Full Software Development Lifecycle: A Case Studycited by 40
  13. 2024TableRAG: Million-Token Table Understanding with Language Modelscited by 40
  14. 2023Data Contamination Through the Lens of Timecited by 39
  15. 2024Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLMcited by 38
  16. 2024WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code Visualizationcited by 38
  17. 2025MLE-Dojo: Interactive Environments for Empowering LLM Agents in Machine Learning Engineeringcited by 38
  18. 2024InsightBench: Evaluating Business Analytics Agents Through Multi-Step Insight Generationcited by 30
  19. 2024ComfyBench: Benchmarking LLM-based Agents in ComfyUI for Autonomously Designing Collaborative AI Systemscited by 30
  20. 2025KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakescited by 30
  21. 2023LLM-Assisted Code Cleaning For Training Accurate Code Generatorscited by 27
  22. 2024MHPP: Exploring the Capabilities and Limitations of Language Models Beyond Basic Code Generationcited by 25
  23. 2023Beyond Static Datasets: A Deep Interaction Approach to LLM Evaluationcited by 24
  24. 2024Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Taskscited by 24
  25. 2025EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Codecited by 24
  26. 2025GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agentscited by 24
  27. 2024Unsupervised Evaluation of Code LLMs with Round-Trip Correctnesscited by 23
  28. 2024CodeBenchGen: Creating Scalable Execution-based Code Generation Benchmarkscited by 23
  29. 2025DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecyclecited by 22
  30. 2024Transforming Wearable Data into Personal Health Insights using Large Language Model Agentscited by 19
  31. 2024Tapilot-Crossing: Benchmarking and Evolving LLMs Towards Interactive Data Analysis Agentscited by 17
  32. 2025BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactionscited by 17
  33. 2025CodeARC: Benchmarking Reasoning Capabilities of LLM Agents for Inductive Program Synthesiscited by 15
  34. 2025LLM/Agent-as-Data-Analyst: A Surveycited by 15
  35. 2025LLM-Based Multi-Agent Blackboard System for Information Discovery in Data Sciencecited by 15
  36. 2026DSGym: A Holistic Framework for Evaluating and Training Data Science Agentscited by 14
  37. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 12
  38. 2025From Reproduction to Replication: Evaluating Research Agents with Progressive Code Maskingcited by 12

50 most cited of 86 All 86 (JSON)