# On the validity of pre-trained transformers for natural language processing in the software engineering domain arXiv: https://arxiv.org/abs/2109.04738 ยท HTML: https://mlsanity.com/paper/2109.04738/ ## Key facts - Cited by 11 papers in the map (data through 14 Oct 2026). - Builds on 33 of its 78 references that are in the map. - First posted on arXiv 10 Sep 2021. ## Builds on (most cited first, 30 of 33) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019) - [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018) - [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013) - [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016) - [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021) - [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019) - [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019) - [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018) - [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019) - [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019) - [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020) - [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019) - [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020) - [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019) - [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017) - [Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](https://mlsanity.com/paper/1904.00962/) (2019) - [ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission](https://mlsanity.com/paper/1904.05342/) (2019) - [FinBERT: Financial Sentiment Analysis with Pre-trained Language Models](https://mlsanity.com/paper/1908.10063/) (2019) - [A Multiscale Visualization of Attention in the Transformer Model](https://mlsanity.com/paper/1906.05714/) (2019) - [code2vec: Learning Distributed Representations of Code](https://mlsanity.com/paper/1803.09473/) (2018) - [Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems](https://mlsanity.com/paper/1805.04508/) (2018) - [Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysis](https://mlsanity.com/paper/1606.04316/) (2016) - [Code and Named Entity Recognition in StackOverflow](https://mlsanity.com/paper/2005.01634/) (2020) - [A Benchmark Study on Sentiment Analysis for Software Engineering Research](https://mlsanity.com/paper/1803.06525/) (2018) - [Sentiment Polarity Detection for Software Development](https://mlsanity.com/paper/1709.02984/) (2017) - [Import2vec - Learning Embeddings for Software Libraries](https://mlsanity.com/paper/1904.03990/) (2019) ## Built on by (most cited first, 11 of 11) - [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023) - [Large Language Models (LLMs) for Source Code Analysis: applications, models and datasets](https://mlsanity.com/paper/2503.17502/) (2025) - [Studying the explanations for the automated prediction of bug and non-bug issues using LIME and SHAP](https://mlsanity.com/paper/2209.07623/) (2022) - [Open-Source AI-based SE Tools: Opportunities and Challenges of Collaborative Software Learning](https://mlsanity.com/paper/2404.06201/) (2024) - [Predicting Issue Types with seBERT](https://mlsanity.com/paper/2205.01335/) (2022) - [Innovative Sentiment Analysis and Prediction of Stock Price Using FinBERT, GPT-4 and Logistic Regression: A Data-Driven Approach](https://mlsanity.com/paper/2412.06837/) (2024) - [Characterizing and Classifying Developer Forum Posts with their Intentions](https://mlsanity.com/paper/2312.14279/) (2023) - [Skill over Scale: The Case for Medium, Domain-Specific Models for SE](https://mlsanity.com/paper/2306.03268/) (2023) - [Identifying Technical Debt and Its Types Across Diverse Software Projects Issues](https://mlsanity.com/paper/2408.09128/) (2024) - [In-Context Code-Text Learning for Bimodal Software Engineering](https://mlsanity.com/paper/2410.18107/) (2024) - [ArchISMiner: A Framework for Automatic Mining of Architectural Issue-Solution Pairs from Online Developer Communities](https://mlsanity.com/paper/2510.21966/) (2025) ## Abstract (arXiv) Transformers are the current state-of-the-art of natural language processing in many domains and are using traction within software engineering research as well. Such models are pre-trained on large amounts of data, usually from the general domain. However, we only have a limited understanding regarding the validity of transformers within the software engineering domain, i.e., how good such models are at understanding words and sentences within a software engineering context and how this improves the state-of-the-art. Within this article, we shed light on this complex, but crucial issue. We compare BERT transformer models trained with software engineering data with transformers based on general domain data in multiple dimensions: their vocabulary, their ability to understand which words are missing, and their performance in classification tasks. Our results show that for tasks that require understanding of the software engineering context, pre-training with software engineering data is valuable, while general domain models are sufficient for general language understanding, also within the software engineering domain.