mlsanityData to 7 Oct 2026

The Vault: A Comprehensive Multilingual Dataset for Advancing Code Understanding and Generation

arXiv 2305.06156 ↗First posted 9 May 2023cs.CL cs.AI cs.PLcited by 20

Not on the map. Why?

Abstract (arXiv)

We present The Vault, a dataset of high-quality code-text pairs in multiple programming languages for training large language models to understand and generate code. We present methods for thoroughly extracting samples that use both rule-based and deep learning-based methods to ensure that they contain high-quality pairs of code and text, resulting in a dataset of 43 million high-quality code-text pairs. Our extensive evaluations on common coding tasks including code generation, code search and code summarization show that when fine-tuning Code Large Language Models on The Vault, such models outperform the same models trained on other datasets such as CodeSearchNet. We also provide detailed analyses of our datasets to assess the effects of various programming languages and docstrings on the performance of such models.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 45 of its 66 references that are in the map. First posted on arXiv 9 May 2023.

Builds on

45 of 66 references.

References · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  9. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  10. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  11. 2021Measuring Coding Challenge Competence With APPScited by 879
  12. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
33 more
  1. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  2. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  3. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  4. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  5. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  6. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 355
  7. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  8. 2020Unsupervised Translation of Programming Languagescited by 262
  9. 2020A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languagescited by 131
  10. 2020PyMT5: multi-mode translation of natural language and Python code with transformerscited by 80
  11. 2019A Neural Model for Generating Natural Language Summaries of Program Subroutinescited by 76
  12. 2021CoTexT: Multi-task Learning with Code-Text Transformercited by 61
  13. 2017A parallel corpus of Python functions and documentation strings for automated code documentation and code generationcited by 56
  14. 2022NatGen: Generative pre-training by "Naturalizing" source codecited by 53
  15. 2020Self-Supervised Contrastive Learning for Code Retrieval and Summarization via Semantic-Preserving Transformationscited by 49
  16. 2019Recommendations for Datasets for Source Code Summarizationcited by 46
  17. 2021Multilingual training for Software Engineeringcited by 43
  18. 2022A Survey of Multi-task Learning in Natural Language Processing: Regarding Task Relatedness and Training Methodscited by 42
  19. 2022SPT-Code: Sequence-to-Sequence Pre-Training for Learning Source Code Representationscited by 35
  20. 2022On the Importance of Building High-quality Training Datasets for Neural Code Searchcited by 30
  21. 2021CodeTrans: Towards Cracking the Language of Silicon's Code Through Self-Supervised Deep Learning and High Performance Computingcited by 28
  22. 2021How could Neural Networks understand Programs?cited by 26
  23. 2020InferCode: Self-Supervised Learning of Code Representations by Predicting Subtreescited by 23
  24. 2021CoDesc: A Large Code-Description Parallel Datasetcited by 23
  25. 2022Are We Building on the Rock? On the Importance of Data Preprocessing for Code Summarizationcited by 22
  26. 2022Code Search: A Survey of Techniques for Finding Codecited by 18
  27. 2020TreeCaps: Tree-Based Capsule Networks for Source Code Processingcited by 16
  28. 2023Better Language Models of Code through Self-Improvementcited by 14
  29. 2020Generating Question Titles for Stack Overflow from Mined Code Snippetscited by 10
  30. 2021Code to Comment Translation: A Comparative Study on Model Effectiveness & Errorscited by 6
  31. 2019SAR: Learning Cross-Language API Mappings with Little Knowledgecited by 5
  32. 2022Towards Using Data-Influence Methods to Detect Noisy Samples in Source Code Corporacited by 4
  33. 2022SOTitle: A Transformer-based Post Title Generation Approach for Stack Overflowcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 7; 2025: 9; 2026: 2.

20 papers.

Sort:

By area:

  1. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  2. 2025SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarioscited by 49
  3. 2024Between Lines of Code: Unraveling the Distinct Patterns of Machine and Human Programmerscited by 27
  4. 2025SWE-Synth: Synthesizing Verifiable Bug-Fix Data to Enable Large Language Models in Resolving Real-World Bugscited by 22
  5. 2023CodeTF: One-stop Transformer Library for State-of-the-art Code LLMscited by 21
  6. 2024On the Impacts of Contexts on Repository-Level Code Generationcited by 21
  7. 2024CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMscited by 18
  8. 2024Building a Large Japanese Web Corpus for Large Language Modelscited by 11
  9. 2025textttDroid: A Resource Suite for AI-Generated Code Detectioncited by 9
  10. 2024XMainframe: A Large Language Model for Mainframe Modernizationcited by 6
  11. 2024Envisioning the Next-Generation AI Coding Assistants: Insights & Proposalscited by 2
  12. 2025Treasure Hunt: Real-time Targeting of the Long Tail using Training-Time Markerscited by 2
8 more
  1. 2025StRuCom: A Novel Dataset of Structured Code Comments in Russiancited by 1
  2. 2025CIDRe: A Reference-Free Multi-Aspect Criterion for Code Comment Quality Measurementcited by 1
  3. 2025Protocode: Prototype-Driven Interpretability for Code Generation in LLMscited by 1
  4. 2024Dopamin: Transformer-based Comment Classifiers through Domain Post-Training and Multi-level Layer Aggregationcited by 0
  5. 2025Which Programming Language and Model Work Best With LLM-as-a-Judge For Code Retrieval?cited by 0
  6. 2025SpareCodeSearch: Searching for Code Context When You Have No Spare GPUcited by 0
  7. 2026Separating Secrets from Placeholders: A Hybrid CNN-CodeBERT Framework for Three-Class Credential Leakage Detectioncited by 0
  8. 2026Scientific Code Search at Scale: A Multi-Domain Dataset and Benchmarkcited by 0