mlsanityData to 14 Oct 2026

JavaBERT: Training a transformer-based model for the Java programming language

arXiv 2110.10404 ↗First posted 20 Oct 2021cs.SE cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Code quality is and will be a crucial factor while developing new software code, requiring appropriate tools to ensure functional and reliable code. Machine learning techniques are still rarely used for software engineering tools, missing out the potential benefits of its application. Natural language processing has shown the potential to process text data regarding a variety of tasks. We argue, that such models can also show similar benefits for software code processing. In this paper, we investigate how models used for natural language processing can be trained upon software code. We introduce a data retrieval pipeline for software code and train a model upon Java software code. The resulting model, JavaBERT, shows a high accuracy on the masked language modeling task showing its potential for software engineering tools.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 11 of its 17 references that are in the map. First posted on arXiv 20 Oct 2021.

Builds on

11 of 17 references.

References · 11

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  8. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  9. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  10. 2021Towards Automating Code Review Activitiescited by 54
  11. 2016Learning a Static Analyzer from Datacited by 7

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 3; 2024: 0; 2025: 1.

5 papers.

  1. 2022Deep Learning Meets Software Engineering: A Survey on Pre-Trained Models of Source Codecited by 21
  2. 2023CrossCodeBench: Benchmarking Cross-Task Generalization of Source Code Modelscited by 16
  3. 2023INSPECT: Intrinsic and Systematic Probing Evaluation for Code Transformerscited by 2
  4. 2023An Exploratory Literature Study on Sharing and Energy Use of Language Models for Source Codecited by 1
  5. 2025Today's Cat Is Tomorrow's Dog: Accounting for Time-Based Changes in the Labels of ML Vulnerability Detection Approachescited by 0