mlsanityData to 14 Oct 2026

BAM! Born-Again Multi-Task Networks for Natural Language Understanding

arXiv 1907.04829 ↗First posted 10 Jul 2019cs.CLcited by 85

Not on the map. Why?

Abstract (arXiv)

It can be challenging to train multi-task neural networks that outperform or even match their single-task counterparts. To help address this, we propose using knowledge distillation where single-task models teach a multi-task model. We enhance this training with teacher annealing, a novel method that gradually transitions the model from distillation to supervised learning, helping the multi-task model surpass its single-task teachers. We evaluate our approach by multi-task fine-tuning BERT on the GLUE benchmark. Our method consistently improves over standard single-task and multi-task training.

Key facts. Cited by 85 papers in the map (data through 14 Oct 2026). First posted on arXiv 10 Jul 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 36; 2022: 16; 2023: 17; 2024: 8; 2025: 6; 2026: 2.

85 papers.

Sort:

By area:

  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  2. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 284
  3. 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
  4. 2023Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learningcited by 115
  5. 2024Knowledge Fusion of Large Language Modelscited by 115
  6. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inferencecited by 96
  7. 2021Multi-Task Self-Training for Learning General Representationscited by 85
  8. 2021Multi-Task Learning in Natural Language Processing: An Overviewcited by 62
  9. 2021Annealing Knowledge Distillationcited by 61
  10. 2021Object DGCNN: 3D Object Detection using Dynamic Graphscited by 58
  11. 2022Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEscited by 58
  12. 2021Universal Representation Learning from Multiple Domains for Few-shot Classificationcited by 56
38 more
  1. 2021PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrievalcited by 50
  2. 2022Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systemscited by 44
  3. 2023FAME-ViL: Multi-Tasking Vision-Language Model for Heterogeneous Fashion Taskscited by 38
  4. 2021The Stem Cell Hypothesis: Dilemma behind Multi-Task Learning with Transformer Encoderscited by 32
  5. 2021MATE-KD: Masked Adversarial TExt, a Companion to Knowledge Distillationcited by 27
  6. 2024UNIC: Universal Classification Models via Multi-teacher Distillationcited by 27
  7. 2022Universal Representations: A Unified Look at Multiple Task and Domain Learningcited by 23
  8. 2021Deep Multi-Task Learning for Joint Localization, Perception, and Predictioncited by 22
  9. 2021Specializing Multilingual Language Models: An Empirical Studycited by 21
  10. 2021Student Surpasses Teacher: Imitation Attack for Black-Box NLP APIscited by 18
  11. 2021Pro-KD: Progressive Distillation by Following the Footsteps of the Teachercited by 17
  12. 2023Towards a Unified Transformer-based Framework for Scene Graph Generation and Human-object Interaction Detectioncited by 15
  13. 2021Towards a Unified Foundation Model: Jointly Pre-Training Transformers on Unpaired Images and Textcited by 14
  14. 2021MergeDistill: Merging Pre-trained Language Models using Distillationcited by 13
  15. 2023The Expressive Power of Tuning Only the Normalization Layerscited by 13
  16. 2021De-identification of Privacy-related Entities in Job Postingscited by 12
  17. 2021Not Far Away, Not So Close: Sample Efficient Nearest Neighbour Data Augmentation via MiniMaxcited by 10
  18. 2021Improving Tree-Structured Decoder Training for Code Generation via Mutual Learningcited by 10
  19. 2022Nearest Neighbor Knowledge Distillation for Neural Machine Translationcited by 10
  20. 2022Can Open-Domain QA Reader Utilize External Knowledge Efficiently like Humans?cited by 10
  21. 2023Preventing Catastrophic Forgetting in Continual Learning of New Natural Language Taskscited by 10
  22. 2021What's in your Head? Emergent Behaviour in Multi-Task Transformer Modelscited by 9
  23. 2022FS-BAN: Born-Again Networks for Domain Generalization Few-Shot Classificationcited by 9
  24. 2025Clinical QA 2.0: Multi-Task Learning for Answer Extraction and Categorizationcited by 9
  25. 2021Multitasking Inhibits Semantic Driftcited by 8
  26. 2022Let the Model Decide its Curriculum for Multitask Learningcited by 8
  27. 2023Visual Exemplar Driven Task-Prompting for Unified Perception in Autonomous Drivingcited by 8
  28. 2021Dual-View Distilled BERT for Sentence Embeddingcited by 7
  29. 2021Language Modelling via Learning to Rankcited by 7
  30. 2022Confidence Based Bidirectional Global Context Aware Training Framework for Neural Machine Translationcited by 7
  31. 2022Mask More and Mask Later: Efficient Pre-training of Masked Language Models by Disentangling the [MASK] Tokencited by 7
  32. 2023Adversarial Multi-task Learning for End-to-end Metaphor Detectioncited by 7
  33. 2024PEMT: Multi-Task Correlation Guided Mixture-of-Experts Enables Parameter-Efficient Transfer Learningcited by 7
  34. 2024ToXCL: A Unified Framework for Toxic Speech Detection and Explanationcited by 7
  35. 2021Latent-Optimized Adversarial Neural Transfer for Sarcasm Detectioncited by 6
  36. 2021Improving Question Answering Performance Using Knowledge Distillation and Active Learningcited by 6
  37. 2022Sentiment-Aware Word and Sentence Level Pre-training for Sentiment Analysiscited by 6
  38. 2022Transformers Go for the LOLs: Generating (Humourous) Titles from Scientific Abstracts End-to-Endcited by 6

50 most cited of 85 All 85 (JSON)