# CodeInsight: A Curated Dataset of Practical Coding Solutions from Stack Overflow

arXiv: https://arxiv.org/abs/2409.16819 · HTML: https://mlsanity.com/paper/2409.16819/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 15 references that are in the map.
- First posted on arXiv 25 Sep 2024.

## Builds on (most cited first, 11 of 11)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [StarCoder: may the source be with you!](https://mlsanity.com/paper/2305.06161/) (2023)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation](https://mlsanity.com/paper/2211.11501/) (2022)
- [Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow](https://mlsanity.com/paper/1805.08949/) (2018)
- [Execution-Based Evaluation for Open-Domain Code Generation](https://mlsanity.com/paper/2212.10481/) (2022)
- [Training and Evaluating a Jupyter Notebook Data Science Assistant](https://mlsanity.com/paper/2201.12901/) (2022)
- [The impact of lexical and grammatical processing on generating code from natural language](https://mlsanity.com/paper/2202.13972/) (2022)

## Built on by (most cited first, 4 of 4)

- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [GiFT: Gibbs Fine-Tuning for Code Generation](https://mlsanity.com/paper/2502.11466/) (2025)
- [Library Hallucinations in LLM-Generated Code: A Risk Analysis Grounded in Developer Queries](https://mlsanity.com/paper/2509.22202/) (2025)
- [From Competition to Collaboration: Designing Sustainable Mechanisms Between LLMs and Online Forums](https://mlsanity.com/paper/2602.04572/) (2026)

## Abstract (arXiv)

We introduce a novel dataset tailored for code generation, aimed at aiding developers in common tasks. Our dataset provides examples that include a clarified intent, code snippets associated, and an average of three related unit tests. It encompasses a range of libraries such as textttPandas, textttNumpy, and textttRegex, along with more than 70 standard libraries in Python code derived from Stack Overflow. Comprising 3,409 crafted examples by Python experts, our dataset is designed for both model finetuning and standalone evaluation. To complete unit tests evaluation, we categorize examples in order to get more fine grained analysis, enhancing the understanding of models' strengths and weaknesses in specific coding tasks. The examples have been refined to reduce data contamination, a process confirmed by the performance of three leading models: Mistral 7B, CodeLLaMa 13B, and Starcoder 15B. We further investigate data-contamination testing GPT-4 performance on a part of our dataset. The benchmark can be accessed at urlhttps://github.com/NathanaelBeau/CodeInsight.