# No More Fine-Tuning? An Experimental Evaluation of Prompt Tuning in Code Intelligence

arXiv: https://arxiv.org/abs/2207.11680 · HTML: https://mlsanity.com/paper/2207.11680/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 63 references that are in the map.
- First posted on arXiv 24 Jul 2022.

## Builds on (most cited first, 30 of 49)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://mlsanity.com/paper/2110.07602/) (2021)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [Multimodal Few-Shot Learning with Frozen Language Models](https://mlsanity.com/paper/2106.13884/) (2021)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [Unified Pre-training for Program Understanding and Generation](https://mlsanity.com/paper/2103.06333/) (2021)
- [Factual Probing Is [MASK]: Learning vs. Learning to Recall](https://mlsanity.com/paper/2104.05240/) (2021)
- [Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks](https://mlsanity.com/paper/1909.03496/) (2019)
- [Unsupervised Translation of Programming Languages](https://mlsanity.com/paper/2006.03511/) (2020)
- [PTR: Prompt Tuning with Rules for Text Classification](https://mlsanity.com/paper/2105.11259/) (2021)
- [PPT: Pre-trained Prompt Tuning for Few-shot Learning](https://mlsanity.com/paper/2109.04332/) (2021)
- [IntelliCode Compose: Code Generation Using Transformer](https://mlsanity.com/paper/2005.08025/) (2020)
- [How Context Affects Language Models' Factual Predictions](https://mlsanity.com/paper/2005.04611/) (2020)
- [SpotTune: Transfer Learning through Adaptive Fine-tuning](https://mlsanity.com/paper/1811.08737/) (2018)

## Built on by (most cited first, 30 of 41)

- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [Fuzz4All: Universal Fuzzing with Large Language Models](https://mlsanity.com/paper/2308.04748/) (2023)
- [Evaluating Instruction-Tuned Large Language Models on Code Comprehension and Generation](https://mlsanity.com/paper/2308.01240/) (2023)
- [RepairLLaMA: Efficient Representations and Fine-Tuned Adapters for Program Repair](https://mlsanity.com/paper/2312.15698/) (2023)
- [Beyond Answers: Transferring Reasoning Capabilities to Smaller LLMs Using Multi-Teacher Knowledge Distillation](https://mlsanity.com/paper/2402.04616/) (2024)
- [Large Language Models and Simple, Stupid Bugs](https://mlsanity.com/paper/2303.11455/) (2023)
- [Facing the Elephant in the Room: Visual Prompt Tuning or Full Finetuning?](https://mlsanity.com/paper/2401.12902/) (2024)
- [Source Code Summarization in the Era of Large Language Models](https://mlsanity.com/paper/2407.07959/) (2024)
- [Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Models](https://mlsanity.com/paper/2308.10462/) (2023)
- [How Reliable Are AI-Generated-Text Detectors? An Assessment Framework Using Evasive Soft Prompts](https://mlsanity.com/paper/2310.05095/) (2023)
- [Astraios: Parameter-Efficient Instruction Tuning Code Large Language Models](https://mlsanity.com/paper/2401.00788/) (2024)
- [Deep Learning for Code Intelligence: Survey, Benchmark and Toolkit](https://mlsanity.com/paper/2401.00288/) (2023)
- [Prompt-based Adaptation in Large-scale Vision Models: A Survey](https://mlsanity.com/paper/2510.13219/) (2025)
- [Benchmarking and Explaining Large Language Model-based Code Generation: A Causality-Centric Approach](https://mlsanity.com/paper/2310.06680/) (2023)
- [Who Evaluates the Evaluators? On Automatic Metrics for Assessing AI-based Offensive Code Generators](https://mlsanity.com/paper/2212.06008/) (2022)
- [When Less is Enough: Positive and Unlabeled Learning Model for Vulnerability Detection](https://mlsanity.com/paper/2308.10523/) (2023)
- [An Extensive Study on Adversarial Attack against Pre-trained Models of Code](https://mlsanity.com/paper/2311.07553/) (2023)
- [ESALE: Enhancing Code-Summary Alignment Learning for Source Code Summarization](https://mlsanity.com/paper/2407.01646/) (2024)
- [On the Usage of Continual Learning for Out-of-Distribution Generalization in Pre-trained Language Models of Code](https://mlsanity.com/paper/2305.04106/) (2023)
- [A Comprehensive Evaluation of Parameter-Efficient Fine-Tuning on Software Engineering Tasks](https://mlsanity.com/paper/2312.15614/) (2023)
- [RAG or Fine-tuning? A Comparative Study on LCMs-based Code Completion in Industry](https://mlsanity.com/paper/2505.15179/) (2025)
- [Testing the Effect of Code Documentation on Large Language Model Code Understanding](https://mlsanity.com/paper/2404.03114/) (2024)
- [Do Prompt Patterns Affect Code Quality? A First Empirical Assessment of ChatGPT-Generated Code](https://mlsanity.com/paper/2504.13656/) (2025)
- [Structure-aware Fine-tuning for Code Pre-trained Models](https://mlsanity.com/paper/2404.07471/) (2024)
- [Contextualized Data-Wrangling Code Generation in Computational Notebooks](https://mlsanity.com/paper/2409.13551/) (2024)
- [Enhancing Code Generation via Bidirectional Comment-Level Mutual Grounding](https://mlsanity.com/paper/2505.07768/) (2025)
- [Solving Data-centric Tasks using Large Language Models](https://mlsanity.com/paper/2402.11734/) (2024)
- [Genetic Auto-prompt Learning for Pre-trained Code Intelligence Language Models](https://mlsanity.com/paper/2403.13588/) (2024)
- [Towards Automatically Addressing Self-Admitted Technical Debt: How Far Are We?](https://mlsanity.com/paper/2308.08943/) (2023)
- [Refining Joint Text and Source Code Embeddings for Retrieval Task with Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2405.04126/) (2024)

## Abstract (arXiv)

Pre-trained models have been shown effective in many code intelligence tasks. These models are pre-trained on large-scale unlabeled corpus and then fine-tuned in downstream tasks. However, as the inputs to pre-training and downstream tasks are in different forms, it is hard to fully explore the knowledge of pre-trained models. Besides, the performance of fine-tuning strongly relies on the amount of downstream data, while in practice, the scenarios with scarce data are common. Recent studies in the natural language processing (NLP) field show that prompt tuning, a new paradigm for tuning, alleviates the above issues and achieves promising results in various NLP tasks. In prompt tuning, the prompts inserted during tuning provide task-specific knowledge, which is especially beneficial for tasks with relatively scarce data. In this paper, we empirically evaluate the usage and effect of prompt tuning in code intelligence tasks. We conduct prompt tuning on popular pre-trained models CodeBERT and CodeT5 and experiment with three code intelligence tasks including defect prediction, code summarization, and code translation. Our experimental results show that prompt tuning consistently outperforms fine-tuning in all three tasks. In addition, prompt tuning shows great potential in low-resource scenarios, e.g., improving the BLEU scores of fine-tuning by more than 26% on average for code summarization. Our results suggest that instead of fine-tuning, we could adapt prompt tuning for code intelligence tasks to achieve better performance, especially when lacking task-specific data.