# CodeLL: A Lifelong Learning Dataset to Support the Co-Evolution of Data and Language Models of Code

arXiv: https://arxiv.org/abs/2312.12492 · HTML: https://mlsanity.com/paper/2312.12492/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 29 references that are in the map.
- First posted on arXiv 20 Dec 2023.

## Builds on (most cited first, 16 of 16)

- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [The Stack: 3 TB of permissively licensed source code](https://mlsanity.com/paper/2211.15533/) (2022)
- [CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks](https://mlsanity.com/paper/2105.12655/) (2021)
- [XLCoST: A Benchmark Dataset for Cross-lingual Code Intelligence](https://mlsanity.com/paper/2206.08474/) (2022)
- [How Often Do Single-Statement Bugs Occur? The ManySStuBs4J Dataset](https://mlsanity.com/paper/1905.13334/) (2019)
- [Do Developers Update Their Library Dependencies? An Empirical Study on the Impact of Security Advisories on Library Migration](https://mlsanity.com/paper/1709.04621/) (2017)
- [CCT5: A Code-Change-Oriented Pre-Trained Model](https://mlsanity.com/paper/2305.10785/) (2023)
- [Megadiff: A Dataset of 600k Java Source Code Changes Categorized by Diff Size](https://mlsanity.com/paper/2108.04631/) (2021)
- [Keeping Pace with Ever-Increasing Data: Towards Continual Learning of Code Intelligence Models](https://mlsanity.com/paper/2302.03482/) (2023)
- [The CodRep Machine Learning on Source Code Competition](https://mlsanity.com/paper/1807.03200/) (2018)
- [How Does API Migration Impact Software Quality and Comprehension? An Empirical Study](https://mlsanity.com/paper/1907.07724/) (2019)
- [Fingerprinting and Building Large Reproducible Datasets](https://mlsanity.com/paper/2306.11391/) (2023)

## Abstract (arXiv)

Motivated by recent work on lifelong learning applications for language models (LMs) of code, we introduce CodeLL, a lifelong learning dataset focused on code changes. Our contribution addresses a notable research gap marked by the absence of a long-term temporal dimension in existing code change datasets, limiting their suitability in lifelong learning scenarios. In contrast, our dataset aims to comprehensively capture code changes across the entire release history of open-source software repositories. In this work, we introduce an initial version of CodeLL, comprising 71 machine-learning-based projects mined from Software Heritage. This dataset enables the extraction and in-depth analysis of code changes spanning 2,483 releases at both the method and API levels. CodeLL enables researchers studying the behaviour of LMs in lifelong fine-tuning settings for learning code changes. Additionally, the dataset can help studying data distribution shifts within software repositories and the evolution of API usages over time.