# On the Effectiveness of Transfer Learning for Code Search

arXiv: https://arxiv.org/abs/2108.05890 · HTML: https://mlsanity.com/paper/2108.05890/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Aug 2021.

## Built on by (most cited first, 6 of 6)

- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [ProCQA: A Large-scale Community-based Programming Question Answering Dataset for Code Search](https://mlsanity.com/paper/2403.16702/) (2024)
- [Improving Automated Program Repair with Domain Adaptation](https://mlsanity.com/paper/2212.11414/) (2022)
- [xASTNN: Improved Code Representations for Industrial Practice](https://mlsanity.com/paper/2303.07104/) (2023)
- [The (In)Effectiveness of Intermediate Task Training For Domain Adaptation and Cross-Lingual Transfer Learning](https://mlsanity.com/paper/2210.01091/) (2022)
- [Multi Language Models for On-the-Fly Syntax Highlighting](https://mlsanity.com/paper/2510.04166/) (2025)

## Abstract (arXiv)

The Transformer architecture and transfer learning have marked a quantum leap in natural language processing, improving the state of the art across a range of text-based tasks. This paper examines how these advancements can be applied to and improve code search. To this end, we pre-train a BERT-based model on combinations of natural language and source code data and fine-tune it on pairs of StackOverflow question titles and code answers. Our results show that the pre-trained models consistently outperform the models that were not pre-trained. In cases where the model was pre-trained on natural language "and" source code data, it also outperforms an information retrieval baseline based on Lucene. Also, we demonstrated that the combined use of an information retrieval-based approach followed by a Transformer leads to the best results overall, especially when searching into a large search pool. Transfer learning is particularly effective when much pre-training data is available and fine-tuning data is limited. We demonstrate that natural language processing models based on the Transformer architecture can be directly applied to source code analysis tasks, such as code search. With the development of Transformer models designed more specifically for dealing with source code data, we believe the results of source code analysis tasks can be further improved.