mlsanityData to 14 Oct 2026

Weighted Training for Cross-Task Learning

arXiv 2105.14095 ↗First posted 28 May 2021cs.LG cs.CL stat.MLcited by 22

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce Target-Aware Weighted Training (TAWT), a weighted training algorithm for cross-task learning based on minimizing a representation-based task distance between the source and target tasks. We show that TAWT is easy to implement, is computationally efficient, requires little hyperparameter tuning, and enjoys non-asymptotic learning-theoretic guarantees. The effectiveness of TAWT is corroborated through extensive experiments with BERT on four sequence tagging tasks in natural language processing (NLP), including part-of-speech (PoS) tagging, chunking, predicate detection, and named entity recognition (NER). As a byproduct, the proposed representation-based task distance allows one to reason in a theoretically principled way about several critical aspects of cross-task learning, such as the choice of the source data and the impact of fine-tuning.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 20 of its 39 references that are in the map. First posted on arXiv 28 May 2021.

Builds on

20 of 39 references.

References · 20

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  5. 2017A Survey on Multi-Task Learningcited by 840
  6. 2018Taskonomy: Disentangling Task Transfer Learningcited by 634
  7. 2020What is being transferred in transfer learning?cited by 368
  8. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 350
  9. 2017Automated Curriculum Learning for Neural Networkscited by 257
  10. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 236
  11. 2020Few-Shot Learning via Learning the Representation, Provablycited by 214
  12. 2015The Benefit of Multitask Representation Learningcited by 195
8 more
  1. 2020On the Theory of Transfer Learning: The Importance of Task Diversitycited by 186
  2. 2020Provable Meta-Learning of Linear Representationscited by 161
  3. 2019Transfer Learning for Nonparametric Classification: Minimax Rate and Adaptive Classifiercited by 65
  4. 2020On the Value of Target Data in Transfer Learningcited by 65
  5. 2021How Fine-Tuning Allows for Effective Meta-Learningcited by 41
  6. 2020Minimax Lower Bounds for Transfer Learning with Linear and One-hidden Layer Neural Networkscited by 30
  7. 2020Foreseeing the Benefits of Incidental Supervisioncited by 7
  8. 2020Meta-learning Transferable Representations with a Single Target Domaincited by 5

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 3; 2023: 9; 2024: 5; 2025: 4.

22 papers.

Sort:

By area:

  1. 2023ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learningcited by 28
  2. 2025CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Mergingcited by 23
  3. 2023Instruction Tuned Models are Quick Learnerscited by 21
  4. 2023Identification of Negative Transfers in Multitask Learning Using Surrogate Modelscited by 20
  5. 2022Active Multi-Task Representation Learningcited by 14
  6. 2022Provable and Efficient Continual Representation Learningcited by 14
  7. 2023On the Trade-off of Intra-/Inter-class Diversity for Supervised Pre-trainingcited by 14
  8. 2023Vision Transformer Adapters for Generalizable Multitask Learningcited by 14
  9. 2023Improving Few-Shot Generalization by Exploring and Exploiting Auxiliary Datacited by 13
  10. 2025Task Arithmetic in Trust Region: A Training-Free Model Merging Approach to Navigate Knowledge Conflictscited by 13
  11. 2021Pareto Navigation Gradient Descent: a First-Order Algorithm for Optimization in Pareto Setcited by 12
  12. 2023Improved Active Multi-Task Representation Learning via Lassocited by 9
10 more
  1. 2022Federated Learning on Adaptively Weighted Nodes by Bilevel Optimizationcited by 6
  2. 2024Data Selection for Transfer Unlearningcited by 5
  3. 2024Deep Transfer Learning: Model Framework and Error Analysiscited by 5
  4. 2023Provable Pathways: Learning Multiple Tasks over Multiple Pathscited by 3
  5. 2023Active Representation Learning for General Task Space with Applications in Roboticscited by 2
  6. 2025Cross-regularization: Adaptive Model Complexity through Validation Gradientscited by 2
  7. 2024Enhancing Molecular Property Prediction with Auxiliary Learning and Task-Specific Adaptationcited by 1
  8. 2024Federated Learning Can Find Friends That Are Advantageouscited by 1
  9. 2024Learning Tree-Structured Composition of Data Augmentationcited by 1
  10. 2025GHaLIB: A Multilingual Framework for Hope Speech Detection in Low-Resource Languagescited by 0