mlsanityData to 7 Oct 2026

On The Cross-Modal Transfer from Natural Language to Code through Adapter Modules

arXiv 2204.08653 ↗First posted 19 Apr 2022cs.SE cs.CL cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Pre-trained neural Language Models (PTLM), such as CodeBERT, are recently used in software engineering as models pre-trained on large source code corpora. Their knowledge is transferred to downstream tasks (e.g. code clone detection) via fine-tuning. In natural language processing (NLP), other alternatives for transferring the knowledge of PTLMs are explored through using adapters, compact, parameter efficient modules inserted in the layers of the PTLM. Although adapters are known to facilitate adapting to many downstream tasks compared to fine-tuning the model that require retraining all of the models' parameters -- which owes to the adapters' plug and play nature and being parameter efficient -- their usage in software engineering is not explored. Here, we explore the knowledge transfer using adapters and based on the Naturalness Hypothesis proposed by Hindle et. al citehindle2016naturalness. Thus, studying the bimodality of adapters for two tasks of cloze test and code clone detection, compared to their benchmarks from the CodeXGLUE platform. These adapters are trained using programming languages and are inserted in a PTLM that is pre-trained on English corpora (N-PTLM). Three programming languages, C/C++, Python, and Java, are studied along with extensive experiments on the best setup used for adapters. Improving the results of the N-PTLM confirms the success of the adapters in knowledge transfer to software engineering, which sometimes are in par with or exceed the results of a PTLM trained on source code; while being more efficient in terms of the number of parameters, memory usage, and inference time. Our results can open new directions to build smaller models for more software engineering tasks. We open source all the scripts and the trained adapters.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 28 of its 36 references that are in the map. First posted on arXiv 19 Apr 2022.

Builds on

28 of 36 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  9. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  10. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  11. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  12. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
16 more
  1. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  2. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  3. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  4. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  5. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  6. 2020A Metric Learning Reality Checkcited by 317
  7. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 239
  8. 2017A Survey of Machine Learning for Big Code and Naturalnesscited by 220
  9. 2021CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Taskscited by 199
  10. 2020UNKs Everywhere: Adapting Multilingual Language Models to New Scriptscited by 113
  11. 2021DOBF: A Deobfuscation Pre-Training Objective for Programming Languagescited by 68
  12. 2021Counter-Interference Adapter for Multilingual Machine Translationcited by 39
  13. 2020Exploring Software Naturalness through Neural Language Modelscited by 32
  14. 2020Generating Accurate Assert Statements for Unit Test Cases using Pretrained Transformerscited by 30
  15. 2021CodeTrans: Towards Cracking the Language of Silicon's Code Through Self-Supervised Deep Learning and High Performance Computingcited by 28
  16. 2021Generating Bug-Fixes Using Pretrained Transformerscited by 18

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 1; 2025: 1.

6 papers.

  1. 2023Review of Large Vision Models and Visual Prompt Engineeringcited by 77
  2. 2023Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Modelscited by 28
  3. 2023A Comprehensive Evaluation of Parameter-Efficient Fine-Tuning on Software Engineering Taskscited by 5
  4. 2024Parameter-Efficient Fine-Tuning of Large Language Models for Unit Test Generation: An Empirical Studycited by 3
  5. 2025SeisMoLLM: Advancing Seismic Monitoring via Cross-modal Transfer with Pre-trained Large Language Modelcited by 3
  6. 2023INSPECT: Intrinsic and Systematic Probing Evaluation for Code Transformerscited by 2