mlsanityData to 7 Oct 2026

To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks

arXiv 1903.05987 ↗First posted 14 Mar 2019cs.CL cs.LGcited by 167

Not on the map. Why?

Abstract (arXiv)

While most previous work has focused on different pretraining objectives and architectures for transfer learning, we ask how to best adapt the pretrained model to a given target task. We focus on the two most common forms of adaptation, feature extraction (where the pretrained weights are frozen), and directly fine-tuning the pretrained model. Our empirical results across diverse NLP tasks with two state-of-the-art models show that the relative performance of fine-tuning vs. feature extraction depends on the similarity of the pretraining and target tasks. We explore possible explanations for this finding and provide a set of adaptation guidelines for the NLP practitioner.

Key facts. Cited by 167 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Mar 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 52; 2022: 33; 2023: 39; 2024: 20; 2025: 14; 2026: 9.

167 papers.

Sort:

By area:

  1. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  2. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  3. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  4. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  5. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 254
  6. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  7. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  8. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  9. 2021An Empirical Investigation of the Role of Pre-training in Lifelong Learningcited by 134
  10. 2022Black-box Prompt Learning for Pre-trained Language Modelscited by 70
  11. 2021Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processingcited by 68
  12. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observationscited by 63
38 more
  1. 2022Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learningcited by 58
  2. 2022Pretraining Without Attentioncited by 51
  3. 2024LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Reportcited by 47
  4. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 46
  5. 2022Annotation Error Detection: Analyzing the Past and Present for a More Coherent Futurecited by 45
  6. 2024SAMWISE: Infusing Wisdom in SAM2 for Text-Driven Video Segmentationcited by 43
  7. 2021A Closer Look at How Fine-tuning Changes BERTcited by 42
  8. 2021Scalable Diverse Model Selection for Accessible Transfer Learningcited by 40
  9. 2022PERFECT: Prompt-free and Efficient Few-shot Learning with Language Modelscited by 40
  10. 2021The Low-Dimensional Linear Geometry of Contextualized Word Representationscited by 39
  11. 2023Analyzing Dataset Annotation Quality Management in the Wildcited by 39
  12. 2024OLoRA: Orthonormal Low-Rank Adaptation of Large Language Modelscited by 39
  13. 2023You Only Prompt Once: On the Capabilities of Prompt Learning on Large Language Models to Tackle Toxic Contentcited by 37
  14. 2021Robust Transfer Learning with Pretrained Language Models through Adapterscited by 35
  15. 20245%>100%: Breaking Performance Shackles of Full Fine-Tuning on Visual Recognition Taskscited by 34
  16. 2021The Stem Cell Hypothesis: Dilemma behind Multi-Task Learning with Transformer Encoderscited by 31
  17. 2024Applied Causal Inference Powered by ML and AIcited by 30
  18. 2024Code Representation Learning At Scalecited by 26
  19. 2021DirectProbe: Studying Representations without Classifierscited by 25
  20. 2023Neural Architecture Search for Parameter-Efficient Fine-tuning of Large Pre-trained Language Modelscited by 25
  21. 2023Advancing Neural Encoding of Portuguese with Transformer Albertina PT-*cited by 24
  22. 2023Parameter-efficient is not sufficient: Exploring Parameter, Memory, and Time Efficient Adapter Tuning for Dense Predictionscited by 24
  23. 2021How Does Fine-tuning Affect the Geometry of Embedding Space: A Case Study on Isotropycited by 23
  24. 2021Control Prefixes for Parameter-Efficient Text Generationcited by 23
  25. 2023Tem-adapter: Adapting Image-Text Pretraining for Video Question Answercited by 22
  26. 2021Large-Scale Contextualised Language Modelling for Norwegiancited by 21
  27. 2021Domain Adaptation with Pre-trained Transformers for Query Focused Abstractive Text Summarizationcited by 21
  28. 2024Deeper Insights Without Updates: The Power of In-Context Learning Over Fine-Tuningcited by 20
  29. 2023Adapter is All You Need for Tuning Visual Taskscited by 19
  30. 2023How to prepare your task head for finetuningcited by 18
  31. 2021skweak: Weak Supervision Made Easy for NLPcited by 17
  32. 2021A Theoretical Analysis of Fine-tuning with Linear Teacherscited by 17
  33. 2023Fine-Tuning Games: Bargaining and Adaptation for General-Purpose Modelscited by 17
  34. 2022Clip-Tuning: Towards Derivative-free Prompt Learning with a Mixture of Rewardscited by 14
  35. 2022Hidden State Variability of Pretrained Language Models Can Guide Computation Reduction for Transfer Learningcited by 13
  36. 2025Understanding the Learning Dynamics of LoRA: A Gradient Flow Perspective on Low-Rank Adaptation in Matrix Factorizationcited by 13
  37. 2021An Empirical Study on Hyperparameter Optimization for Fine-Tuning Pre-trained Language Modelscited by 12
  38. 2023Efficient Federated Prompt Tuning for Black-box Large Pre-trained Modelscited by 12

50 most cited of 167 All 167 (JSON)