mlsanityData to 14 Oct 2026

Robustly Optimized and Distilled Training for Natural Language Understanding

arXiv 2103.08809 ↗First posted 16 Mar 2021cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

In this paper, we explore multi-task learning (MTL) as a second pretraining step to learn enhanced universal language representation for transformer language models. We use the MTL enhanced representation across several natural language understanding tasks to improve performance and generalization. Moreover, we incorporate knowledge distillation (KD) in MTL to further boost performance and devise a KD variant that learns effectively from multiple teachers. By combining MTL and KD, we propose Robustly Optimized and Distilled (ROaD) modeling framework. We use ROaD together with the ELECTRA model to obtain state-of-the-art results for machine reading comprehension and natural language inference.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 19 of its 22 references that are in the map. First posted on arXiv 16 Mar 2021.

Builds on

19 of 22 references.

References · 19

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  7. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  8. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  9. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  10. 2015Teaching Machines to Read and Comprehendcited by 1,049
  11. 2019When Does Label Smoothing Help?cited by 988
  12. 2017A Survey on Multi-Task Learningcited by 840
7 more
  1. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 426
  2. 2018QuAC : Question Answering in Contextcited by 381
  3. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 350
  4. 2016NewsQA: A Machine Comprehension Datasetcited by 331
  5. 2020Retrospective Reader for Machine Reading Comprehensioncited by 86
  6. 2019BAM! Born-Again Multi-Task Networks for Natural Language Understandingcited by 85
  7. 2018Large-Scale QA-SRL Parsingcited by 45

Built on by

Cited by year

Citing papers per year: 2021: 1.

1 paper.

  1. 2021Decoupled Transformer for Scalable Inference in Open-domain Question Answeringcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023A Comparative Analysis of Task-Agnostic Distillation Methods for Compressing Transformer Language Models
  2. 2024DistiLLM: Towards Streamlined Distillation for Large Language Models
  3. 2023ERNIE 3.0 Tiny: Frustratingly Simple Method to Improve Task-Agnostic Distillation Generalization
  4. 2023A Cohesive Distillation Architecture for Neural Language Models
  5. 2023A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Training
  6. 2024Multi-Stage Balanced Distillation: Addressing Long-Tail Challenges in Sequence-Level Knowledge Distillation
  7. 2025Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
  8. 2026From Images to Words: Efficient Cross-Modal Knowledge Distillation to Language Models from Black-box Teachers