mlsanityData to 7 Oct 2026

Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow

arXiv 1805.08949 ↗First posted 23 May 2018cs.CL cs.SEcited by 165

Not on the map. Why?

Abstract (arXiv)

For tasks like code synthesis from natural language, code retrieval, and code summarization, data-driven models have shown great promise. However, creating these models require parallel data between natural language (NL) and code with fine-grained alignments. Stack Overflow (SO) is a promising source to create such a data set: the questions are diverse and most of them have corresponding answers with high-quality code snippets. However, existing heuristic methods (e.g., pairing the title of a post with the code in the accepted answer) are limited both in their coverage and the correctness of the NL-code pairs obtained. In this paper, we propose a novel method to mine high-quality aligned data from SO using two sets of features: hand-crafted features considering the structure of the extracted snippets, and correspondence features obtained by training a probabilistic model to capture the correlation between NL and code using neural networks. These features are fed into a classifier that determines the quality of mined NL-code pairs. Experiments using Python and Java as test beds show that the proposed method greatly expands coverage and accuracy over existing mining methods, even when using only a small number of labeled examples. Further, we find that reasonable results are achieved even when training the classifier on one language and testing on another, showing promise for scaling NL-code mining to a wide variety of programming languages beyond those for which we are able to annotate data.

Key facts. Cited by 165 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 May 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 17; 2022: 34; 2023: 24; 2024: 49; 2025: 31; 2026: 10.

165 papers.

Sort:

By area:

  1. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  2. 2024A Survey on Large Language Models for Code Generationcited by 646
  3. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  4. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  5. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 278
  6. 2024CodeS: Towards Building Open-source Language Models for Text-to-SQLcited by 183
  7. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  8. 2024SciCode: A Research Coding Benchmark Curated by Scientistscited by 172
  9. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 166
  10. 2022Multi-lingual Evaluation of Code Generation Modelscited by 158
  11. 2022DocPrompting: Generating Code by Retrieving the Docscited by 151
  12. 2022Large Language Models Meet NL2Code: A Surveycited by 135
38 more
  1. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 132
  2. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 124
  3. 2022Natural Language to Code Translation with Executioncited by 122
  4. 2023CodeBERTScore: Evaluating Code Generation with Pretrained Models of Codecited by 101
  5. 2022Execution-Based Evaluation for Open-Domain Code Generationcited by 89
  6. 2021CoSQA: 20,000+ Web Queries for Code Search and Question Answeringcited by 74
  7. 2023Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generationcited by 72
  8. 2024EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositoriescited by 71
  9. 2024DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositoriescited by 71
  10. 2022Out of the BLEU: how should we assess quality of the Code Generation models?cited by 65
  11. 2022Parsel: Algorithmic Reasoning with Language Models by Composing Decompositionscited by 62
  12. 2024What's Wrong with Your Code Generated by Large Language Models? An Extensive Studycited by 52
  13. 2023ICE-Score: Instructing Large Language Models to Evaluate Codecited by 50
  14. 2023xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrievalcited by 46
  15. 2023SelfEvolve: A Code Evolution Framework via Large Language Modelscited by 43
  16. 2021SpreadsheetCoder: Formula Prediction from Semi-structured Contextcited by 41
  17. 2022MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languagescited by 41
  18. 2024CodeJudge: Evaluating Code Generation with Large Language Modelscited by 41
  19. 2024LAB: Large-Scale Alignment for ChatBotscited by 39
  20. 2025LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarkscited by 34
  21. 2024EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluationscited by 33
  22. 2023Measuring The Impact Of Programming Language Distributioncited by 32
  23. 2024Evaluating Language Models as Synthetic Data Generatorscited by 32
  24. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Userscited by 32
  25. 2022Execution-based Evaluation for Data Science Code Generation Modelscited by 31
  26. 2023CodeFusion: A Pre-trained Diffusion Model for Code Generationcited by 31
  27. 2022On the Importance of Building High-quality Training Datasets for Neural Code Searchcited by 30
  28. 2025MergeBench: A Benchmark for Merging Domain-Specialized LLMscited by 30
  29. 2022ERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming Languagescited by 29
  30. 2023Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Modelscited by 28
  31. 2024CodeS: Natural Language to Code Repository via Multi-Layer Sketchcited by 28
  32. 2024Revisiting Code Similarity Evaluation with Abstract Syntax Tree Edit Distancecited by 26
  33. 2025ProjectEval: A Benchmark for Programming Agents Automated Evaluation on Project-Level Code Generationcited by 26
  34. 2024Are Large Language Models Reliable Argument Quality Annotators?cited by 24
  35. 2024HumanEvalComm: Benchmarking the Communication Competence of Code Generation for LLMs and LLM Agentcited by 24
  36. 2024Automatically Generating UI Code from Screenshot: A Divide-and-Conquer-Based Approachcited by 24
  37. 2021CoDesc: A Large Code-Description Parallel Datasetcited by 23
  38. 2021A Survey on Machine Learning Techniques for Source Code Analysiscited by 23

50 most cited of 165 All 165 (JSON)