mlsanityData to 14 Oct 2026

Distillation of encoder-decoder transformers for sequence labelling

arXiv 2302.05454 ↗First posted 10 Feb 2023cs.CL cs.IRcited by 3

Not on the map. Why?

Abstract (arXiv)

Driven by encouraging results on a wide range of tasks, the field of NLP is experiencing an accelerated race to develop bigger language models. This race for bigger models has also underscored the need to continue the pursuit of practical distillation approaches that can leverage the knowledge acquired by these big models in a compute-efficient manner. Having this goal in mind, we build on recent work to propose a hallucination-free framework for sequence tagging that is especially suited for distillation. We show empirical results of new state-of-the-art performance across multiple sequence labelling datasets and validate the usefulness of this framework for distilling a large model in a few-shot learning scenario.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 25 of its 38 references that are in the map. First posted on arXiv 10 Feb 2023.

Builds on

25 of 38 references.

References · 25

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  8. 2017Decoupled Weight Decay Regularizationcited by 1,831
  9. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  10. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  11. 2016Neural Architectures for Named Entity Recognitioncited by 550
  12. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
13 more
  1. 2018Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfacescited by 377
  2. 2020Autoregressive Entity Retrievalcited by 334
  3. 2021Structured Prediction as Translation between Augmented Natural Languagescited by 183
  4. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  5. 2020MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmarkcited by 158
  6. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 153
  7. 2021A Unified Generative Framework for Various NER Subtaskscited by 141
  8. 2018Cross-Lingual Transfer Learning for Multilingual Task Oriented Dialogcited by 125
  9. 2021LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5cited by 104
  10. 2020Pre-trained Summarization Distillationcited by 71
  11. 2020XtremeDistil: Multi-stage Distillation for Massive Multilingual Modelscited by 38
  12. 2020Augmented Natural Language for Generative Sequence Labelingcited by 29
  13. 2022Transforming Sequence Tagging Into A Seq2Seq Taskcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 0; 2026: 1.

3 papers.

  1. 2024Transformers and Language Models in Form Understanding: A Comprehensive Review of Scanned Document Analysiscited by 6
  2. 2024HyperLoader: Integrating Hypernetwork-Based LoRA and Adapter Layers into Multi-Task Transformers for Sequence Labellingcited by 6
  3. 2026Toward Federated Large Language Models in Medicine: A Parameter-Efficient Framework for Privacy-Preserving, Multi-Institutional Adaptationcited by 1