mlsanityData to 7 Oct 2026

Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code

arXiv 2003.07914 ↗First posted 17 Mar 2020cs.SEcited by 53

Not on the map. Why?

Abstract (arXiv)

Statistical language modeling techniques have successfully been applied to large source code corpora, yielding a variety of new software development tools, such as tools for code suggestion, improving readability, and API migration. A major issue with these techniques is that code introduces new vocabulary at a far higher rate than natural language, as new identifier names proliferate. Both large vocabularies and out-of-vocabulary issues severely affect Neural Language Models (NLMs) of source code, degrading their performance and rendering them unable to scale. In this paper, we address this issue by: 1) studying how various modelling choices impact the resulting vocabulary on a large-scale corpus of 13,362 projects; 2) presenting an open vocabulary source code NLM that can scale to such a corpus, 100 times larger than in previous work; and 3) showing that such models outperform the state of the art on three distinct code corpora (Java, C, Python). To our knowledge, these are the largest NLMs for code that have been reported. All datasets, code, and trained models used in this work are publicly available.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 11; 2023: 12; 2024: 7; 2025: 7; 2026: 2.

53 papers.

Sort:

By area:

  1. 2021Program Synthesis with Large Language Modelscited by 3,509
  2. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  3. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  4. 2023An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generationcited by 193
  5. 2022ReACC: A Retrieval-Augmented Code Completion Frameworkcited by 117
  6. 2021Log-based Anomaly Detection Without Log Parsingcited by 59
  7. 2021PalmTree: Learning an Assembly Language Model for Instruction Embeddingcited by 49
  8. 2022CodeFill: Multi-token Code Completion by Jointly Learning from Structure and Naming Sequencescited by 49
  9. 2024REPOFUSE: Repository-Level Code Completion with Fused Dual Contextcited by 32
  10. 2021Code Structure Guided Transformer for Source Code Summarizationcited by 29
  11. 2021On Multi-Modal Learning of Editing Source Codecited by 25
  12. 2021Learning to Complete Code with Sketchescited by 23
38 more
  1. 2021A Survey on Machine Learning Techniques for Source Code Analysiscited by 23
  2. 2023Deep Learning for Code Intelligence: Survey, Benchmark and Toolkitcited by 22
  3. 2023Guiding Language Models of Code with Global Context using Monitorscited by 20
  4. 2025Correctness Assessment of Code Generated by Large Language Models Using Internal Representationscited by 17
  5. 2023Better Context Makes Better Code Language Models: A Case Study on Function Call Argument Completioncited by 14
  6. 2023Toward a Theory of Causation for Interpreting Neural Code Modelscited by 12
  7. 2023Static Code Analysis in the AI Era: An In-depth Exploration of the Concept, Function, and Potential of Intelligent Code Analysis Agentscited by 12
  8. 2024Studying LLM Performance on Closed- and Open-source Datacited by 11
  9. 2024An Investigation into Misuse of Java Security APIs by Large Language Modelscited by 11
  10. 2022Syntax-Aware On-the-Fly Code Completioncited by 10
  11. 2024Large Language Model Evaluation Via Multi AI Agents: Preliminary resultscited by 10
  12. 2024Towards More Trustworthy and Interpretable LLMs for Code through Syntax-Grounded Explanationscited by 10
  13. 2024RAMBO: Enhancing RAG-based Repository-Level Method Body Completioncited by 10
  14. 2023CodeBPE: Investigating Subtokenization Options for Large Language Model Pretraining on Source Codecited by 9
  15. 2021SYNFIX: Automatically Fixing Syntax Errors using Compiler Diagnosticscited by 8
  16. 2021Capturing Structural Locality in Non-parametric Language Modelscited by 8
  17. 2021Recommending Metamodel Concepts during Modeling Activities with Pre-Trained Language Modelscited by 7
  18. 2026AlignCoder: Aligning Retrieval with Target Intent for Repository-Level Code Completioncited by 7
  19. 2021Program Merge Conflict Resolution via Neural Transformerscited by 5
  20. 2022A Neural Network Architecture for Program Understanding Inspired by Human Behaviorscited by 5
  21. 2022No Word Embedding Model Is Perfect: Evaluating the Representation Accuracy for Social Bias in the Mediacited by 5
  22. 2022A Survey of Deep Learning Models for Structural Code Understandingcited by 4
  23. 2021Learning to Extend Program Graphs to Work-in-Progress Codecited by 3
  24. 2022Heterogeneous Graph Neural Networks for Software Effort Estimationcited by 3
  25. 2023CCRep: Learning Code Change Representations via Pre-Trained Code Model and Query Backcited by 3
  26. 2025The Fools are Certain; the Wise are Doubtful: Exploring LLM Confidence in Code Completioncited by 3
  27. 2025How Different Tokenization Algorithms Impact LLMs and Transformer Models for Binary Code Analysiscited by 3
  28. 2022JEMMA: An Extensible Java Dataset for ML4Code Applicationscited by 2
  29. 2023ARIST: An Effective API Argument Recommendation Approachcited by 2
  30. 2023INSPECT: Intrinsic and Systematic Probing Evaluation for Code Transformerscited by 2
  31. 2022LAMNER: Code Comment Generation Using Character Language Model and Named Entity Recognitioncited by 1
  32. 2022Cross-Domain Evaluation of a Deep Learning-Based Type Inference Systemcited by 1
  33. 2023An Exploratory Literature Study on Sharing and Energy Use of Language Models for Source Codecited by 1
  34. 2025Previously on... Automating Code Reviewcited by 1
  35. 2021DeepMutants: Training neural bug detectors with contextual mutationscited by 0
  36. 2023Skill over Scale: The Case for Medium, Domain-Specific Models for SEcited by 0
  37. 2024Lost in Transmission: An Information-Theoretic Account of Unsupervised Software Traceabilitycited by 0
  38. 2025Mapping the Trust Terrain: LLMs in Software Engineering -- Insights and Perspectivescited by 0

50 most cited of 53 All 53 (JSON)