# JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generation

arXiv: https://arxiv.org/abs/1910.02216 · HTML: https://mlsanity.com/paper/1910.02216/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Oct 2019.

## Built on by (most cited first, 30 of 49)

- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [InCoder: A Generative Model for Code Infilling and Synthesis](https://mlsanity.com/paper/2204.05999/) (2022)
- [DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation](https://mlsanity.com/paper/2211.11501/) (2022)
- [CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution](https://mlsanity.com/paper/2401.03065/) (2024)
- [LEVER: Learning to Verify Language-to-Code Generation with Execution](https://mlsanity.com/paper/2302.08468/) (2023)
- [InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback](https://mlsanity.com/paper/2306.14898/) (2023)
- [Large Language Models Meet NL2Code: A Survey](https://mlsanity.com/paper/2212.09420/) (2022)
- [Document Understanding Dataset and Evaluation (DUDE)](https://mlsanity.com/paper/2305.08455/) (2023)
- [Execution-Based Evaluation for Open-Domain Code Generation](https://mlsanity.com/paper/2212.10481/) (2022)
- [Natural Language to Code Generation in Interactive Data Science Notebooks](https://mlsanity.com/paper/2212.09248/) (2022)
- [If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents](https://mlsanity.com/paper/2401.00812/) (2024)
- [Out of the BLEU: how should we assess quality of the Code Generation models?](https://mlsanity.com/paper/2208.03133/) (2022)
- [xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval](https://mlsanity.com/paper/2303.03004/) (2023)
- [Data Contamination Through the Lens of Time](https://mlsanity.com/paper/2310.10628/) (2023)
- [Execution-based Evaluation for Data Science Code Generation Models](https://mlsanity.com/paper/2211.09374/) (2022)
- [Training and Evaluating a Jupyter Notebook Data Science Assistant](https://mlsanity.com/paper/2201.12901/) (2022)
- [Text2Cypher: Bridging Natural Language and Graph Databases](https://mlsanity.com/paper/2412.10064/) (2024)
- [Toward Secure Code Generation: Bridging Correctness and Security via Task-Adaptive Vulnerability Modeling and Execution-Based Benchmarking](https://mlsanity.com/paper/2407.02395/) (2024)
- [MHPP: Exploring the Capabilities and Limitations of Language Models Beyond Basic Code Generation](https://mlsanity.com/paper/2405.11430/) (2024)
- [Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks](https://mlsanity.com/paper/2403.04814/) (2024)
- [Unsupervised Evaluation of Code LLMs with Round-Trip Correctness](https://mlsanity.com/paper/2402.08699/) (2024)
- [L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models](https://mlsanity.com/paper/2309.17446/) (2023)
- [Deep Learning for Code Intelligence: Survey, Benchmark and Toolkit](https://mlsanity.com/paper/2401.00288/) (2023)
- [Benchmarks and Metrics for Evaluations of Code Generation: A Critical Review](https://mlsanity.com/paper/2406.12655/) (2024)
- [FrontendBench: A Benchmark for Evaluating LLMs on Front-End Development via Automatic Evaluation](https://mlsanity.com/paper/2506.13832/) (2025)
- [Python Code Generation by Asking Clarification Questions](https://mlsanity.com/paper/2212.09885/) (2022)
- [Enhancing Large Language Models for Secure Code Generation: A Dataset-driven Study on Vulnerability Mitigation](https://mlsanity.com/paper/2310.16263/) (2023)
- [Who Evaluates the Evaluators? On Automatic Metrics for Assessing AI-based Offensive Code Generators](https://mlsanity.com/paper/2212.06008/) (2022)
- [Software Development Life Cycle Perspective: A Survey of Benchmarks for Code Large Language Models and Agents](https://mlsanity.com/paper/2505.05283/) (2025)

## Abstract (arXiv)

Interactive programming with interleaved code snippet cells and natural language markdown is recently gaining popularity in the form of Jupyter notebooks, which accelerate prototyping and collaboration. To study code generation conditioned on a long context history, we present JuICe, a corpus of 1.5 million examples with a curated test set of 3.7K instances based on online programming assignments. Compared with existing contextual code generation datasets, JuICe provides refined human-curated data, open-domain code, and an order of magnitude more training data. Using JuICe, we train models for two tasks: (1) generation of the API call sequence in a code cell, and (2) full code cell generation, both conditioned on the NL-Code history up to a particular code cell. Experiments using current baseline code generation models show that both context and distant supervision aid in generation, and that the dataset is challenging for current systems.