# Characterizing and Avoiding Negative Transfer

arXiv: https://arxiv.org/abs/1811.09751 · HTML: https://mlsanity.com/paper/1811.09751/

## Key facts

- Cited by 229 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Nov 2018.

## Built on by (most cited first, 30 of 229)

- [Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks](https://mlsanity.com/paper/2106.04489/) (2021)
- [Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study](https://mlsanity.com/paper/2105.03790/) (2021)
- [GFT: Graph Foundation Model with Transferable Tree Vocabulary](https://mlsanity.com/paper/2411.06070/) (2024)
- [Sensitivity-Aware Visual Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2303.08566/) (2023)
- [The Shape of Learning Curves: a Review](https://mlsanity.com/paper/2103.10948/) (2021)
- [Uncertainty-Aware Unsupervised Domain Adaptation in Object Detection](https://mlsanity.com/paper/2103.00236/) (2021)
- [Distribution Matching for Multi-Task Learning of Classification Tasks: a Large-Scale Study on Faces & Beyond](https://mlsanity.com/paper/2401.01219/) (2024)
- [Transferability in Deep Learning: A Survey](https://mlsanity.com/paper/2201.05867/) (2022)
- [Multi-VALUE: A Framework for Cross-Dialectal English NLP](https://mlsanity.com/paper/2212.08011/) (2022)
- [Frustratingly Easy Transferability Estimation](https://mlsanity.com/paper/2106.09362/) (2021)
- [ID and OOD Performance Are Sometimes Inversely Correlated on Real-world Datasets](https://mlsanity.com/paper/2209.00613/) (2022)
- [RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs](https://mlsanity.com/paper/2407.02552/) (2024)
- [Robust and Efficient Medical Imaging with Self-Supervision](https://mlsanity.com/paper/2205.09723/) (2022)
- [Adapting Off-the-Shelf Source Segmenter for Target Medical Image Segmentation](https://mlsanity.com/paper/2106.12497/) (2021)
- [FedCG: Leverage Conditional GAN for Protecting Privacy and Maintaining Competitive Performance in Federated Learning](https://mlsanity.com/paper/2111.08211/) (2021)
- [SLTUNET: A Simple Unified Model for Sign Language Translation](https://mlsanity.com/paper/2305.01778/) (2023)
- [MosaicOS: A Simple and Effective Use of Object-Centric Images for Long-Tailed Object Detection](https://mlsanity.com/paper/2102.08884/) (2021)
- [Addressing Negative Transfer in Diffusion Models](https://mlsanity.com/paper/2306.00354/) (2023)
- [Leveraging sparse and shared feature activations for disentangled representation learning](https://mlsanity.com/paper/2304.07939/) (2023)
- [Model Spider: Learning to Rank Pre-Trained Models Efficiently](https://mlsanity.com/paper/2306.03900/) (2023)
- [Why Does Little Robustness Help? A Further Step Towards Understanding Adversarial Transferability](https://mlsanity.com/paper/2307.07873/) (2023)
- [High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning](https://mlsanity.com/paper/2203.01311/) (2022)
- [Recent Advances in Generative AI and Large Language Models: Current Status, Challenges, and Perspectives](https://mlsanity.com/paper/2407.14962/) (2024)
- [How Well Does Self-Supervised Pre-Training Perform with Streaming Data?](https://mlsanity.com/paper/2104.12081/) (2021)
- [Privacy-preserving Federated Adversarial Domain Adaption over Feature Groups for Interpretability](https://mlsanity.com/paper/2111.10934/) (2021)
- [Disentangled Federated Learning for Tackling Attributes Skew via Invariant Aggregation and Diversity Transferring](https://mlsanity.com/paper/2206.06818/) (2022)
- [ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learning](https://mlsanity.com/paper/2301.12618/) (2023)
- [Deep Transfer Learning for Automatic Speech Recognition: Towards Better Generalization](https://mlsanity.com/paper/2304.14535/) (2023)
- [Should We Be Pre-training? An Argument for End-task Aware Training as an Alternative](https://mlsanity.com/paper/2109.07437/) (2021)
- [Focus on the Common Good: Group Distributional Robustness Follows](https://mlsanity.com/paper/2110.02619/) (2021)

## Abstract (arXiv)

When labeled data is scarce for a specific target task, transfer learning often offers an effective solution by utilizing data from a related source task. However, when transferring knowledge from a less related source, it may inversely hurt the target performance, a phenomenon known as negative transfer. Despite its pervasiveness, negative transfer is usually described in an informal manner, lacking rigorous definition, careful analysis, or systematic treatment. This paper proposes a formal definition of negative transfer and analyzes three important aspects thereof. Stemming from this analysis, a novel technique is proposed to circumvent negative transfer by filtering out unrelated source data. Based on adversarial networks, the technique is highly generic and can be applied to a wide range of transfer learning algorithms. The proposed approach is evaluated on six state-of-the-art deep transfer methods via experiments on four benchmark datasets with varying levels of difficulty. Empirically, the proposed method consistently improves the performance of all baseline methods and largely avoids negative transfer, even when the source data is degenerate.