# Weighted Training for Cross-Task Learning

arXiv: https://arxiv.org/abs/2105.14095 · HTML: https://mlsanity.com/paper/2105.14095/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 39 references that are in the map.
- First posted on arXiv 28 May 2021.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [What is being transferred in transfer learning?](https://mlsanity.com/paper/2008.11687/) (2020)
- [Multi-Task Deep Neural Networks for Natural Language Understanding](https://mlsanity.com/paper/1901.11504/) (2019)
- [Automated Curriculum Learning for Neural Networks](https://mlsanity.com/paper/1704.03003/) (2017)
- [The Natural Language Decathlon: Multitask Learning as Question Answering](https://mlsanity.com/paper/1806.08730/) (2018)
- [Few-Shot Learning via Learning the Representation, Provably](https://mlsanity.com/paper/2002.09434/) (2020)
- [The Benefit of Multitask Representation Learning](https://mlsanity.com/paper/1505.06279/) (2015)
- [On the Theory of Transfer Learning: The Importance of Task Diversity](https://mlsanity.com/paper/2006.11650/) (2020)
- [Provable Meta-Learning of Linear Representations](https://mlsanity.com/paper/2002.11684/) (2020)
- [Transfer Learning for Nonparametric Classification: Minimax Rate and Adaptive Classifier](https://mlsanity.com/paper/1906.02903/) (2019)
- [On the Value of Target Data in Transfer Learning](https://mlsanity.com/paper/2002.04747/) (2020)
- [How Fine-Tuning Allows for Effective Meta-Learning](https://mlsanity.com/paper/2105.02221/) (2021)
- [Minimax Lower Bounds for Transfer Learning with Linear and One-hidden Layer Neural Networks](https://mlsanity.com/paper/2006.10581/) (2020)
- [Foreseeing the Benefits of Incidental Supervision](https://mlsanity.com/paper/2006.05500/) (2020)
- [Meta-learning Transferable Representations with a Single Target Domain](https://mlsanity.com/paper/2011.01418/) (2020)

## Built on by (most cited first, 22 of 22)

- [ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learning](https://mlsanity.com/paper/2301.12618/) (2023)
- [CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging](https://mlsanity.com/paper/2505.06977/) (2025)
- [Instruction Tuned Models are Quick Learners](https://mlsanity.com/paper/2306.05539/) (2023)
- [Identification of Negative Transfers in Multitask Learning Using Surrogate Models](https://mlsanity.com/paper/2303.14582/) (2023)
- [Active Multi-Task Representation Learning](https://mlsanity.com/paper/2202.00911/) (2022)
- [Provable and Efficient Continual Representation Learning](https://mlsanity.com/paper/2203.02026/) (2022)
- [On the Trade-off of Intra-/Inter-class Diversity for Supervised Pre-training](https://mlsanity.com/paper/2305.12224/) (2023)
- [Vision Transformer Adapters for Generalizable Multitask Learning](https://mlsanity.com/paper/2308.12372/) (2023)
- [Improving Few-Shot Generalization by Exploring and Exploiting Auxiliary Data](https://mlsanity.com/paper/2302.00674/) (2023)
- [Task Arithmetic in Trust Region: A Training-Free Model Merging Approach to Navigate Knowledge Conflicts](https://mlsanity.com/paper/2501.15065/) (2025)
- [Pareto Navigation Gradient Descent: a First-Order Algorithm for Optimization in Pareto Set](https://mlsanity.com/paper/2110.08713/) (2021)
- [Improved Active Multi-Task Representation Learning via Lasso](https://mlsanity.com/paper/2306.02556/) (2023)
- [Federated Learning on Adaptively Weighted Nodes by Bilevel Optimization](https://mlsanity.com/paper/2207.10751/) (2022)
- [Data Selection for Transfer Unlearning](https://mlsanity.com/paper/2405.10425/) (2024)
- [Deep Transfer Learning: Model Framework and Error Analysis](https://mlsanity.com/paper/2410.09383/) (2024)
- [Provable Pathways: Learning Multiple Tasks over Multiple Paths](https://mlsanity.com/paper/2303.04338/) (2023)
- [Active Representation Learning for General Task Space with Applications in Robotics](https://mlsanity.com/paper/2306.08942/) (2023)
- [Cross-regularization: Adaptive Model Complexity through Validation Gradients](https://mlsanity.com/paper/2506.19755/) (2025)
- [Enhancing Molecular Property Prediction with Auxiliary Learning and Task-Specific Adaptation](https://mlsanity.com/paper/2401.16299/) (2024)
- [Federated Learning Can Find Friends That Are Advantageous](https://mlsanity.com/paper/2402.05050/) (2024)
- [Learning Tree-Structured Composition of Data Augmentation](https://mlsanity.com/paper/2408.14381/) (2024)
- [GHaLIB: A Multilingual Framework for Hope Speech Detection in Low-Resource Languages](https://mlsanity.com/paper/2512.22705/) (2025)

## Abstract (arXiv)

In this paper, we introduce Target-Aware Weighted Training (TAWT), a weighted training algorithm for cross-task learning based on minimizing a representation-based task distance between the source and target tasks. We show that TAWT is easy to implement, is computationally efficient, requires little hyperparameter tuning, and enjoys non-asymptotic learning-theoretic guarantees. The effectiveness of TAWT is corroborated through extensive experiments with BERT on four sequence tagging tasks in natural language processing (NLP), including part-of-speech (PoS) tagging, chunking, predicate detection, and named entity recognition (NER). As a byproduct, the proposed representation-based task distance allows one to reason in a theoretically principled way about several critical aspects of cross-task learning, such as the choice of the source data and the impact of fine-tuning.