mlsanityData to 14 Oct 2026

Improving BERT Fine-Tuning via Self-Ensemble and Self-Distillation

arXiv 2002.10345 ↗First posted 24 Feb 2020cs.CL cs.LGcited by 25

Not on the map. Why?

Abstract (arXiv)

Fine-tuning pre-trained language models like BERT has become an effective way in NLP and yields state-of-the-art results on many downstream tasks. Recent studies on adapting BERT to new tasks mainly focus on modifying the model structure, re-designing the pre-train tasks, and leveraging external data and knowledge. The fine-tuning strategy itself has yet to be fully explored. In this paper, we improve the fine-tuning of BERT with two effective mechanisms: self-ensemble and self-distillation. The experiments on text classification and natural language inference tasks show our proposed methods can significantly improve the adaption of BERT without any external data or knowledge.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). First posted on arXiv 24 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 7; 2022: 9; 2023: 2; 2024: 6; 2025: 1.

25 papers.

Sort:

By area:

  1. 2023Speculative Decoding with Big Little Decodercited by 164
  2. 2022BiBERT: Accurate Fully Binarized BERTcited by 73
  3. 2022Unified Implicit Neural Stylizationcited by 67
  4. 2021Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Datacited by 31
  5. 2022Knowledge Distillation for Federated Learning: a Practical Guidecited by 31
  6. 2021MirrorWiC: On Eliciting Word-in-Context Representations from Pretrained Language Modelscited by 18
  7. 2021Alternative Input Signals Ease Transfer in Multilingual Machine Translationcited by 14
  8. 2022Multi-CLS BERT: An Efficient Alternative to Traditional Ensemblingcited by 14
  9. 2022Cluster & Tune: Boost Cold Start Performance in Text Classificationcited by 13
  10. 2021Neighborhood Consensus Contrastive Learning for Backward-Compatible Representationcited by 8
  11. 2024Large Language Models for Relevance Judgment in Product Searchcited by 8
  12. 2021AT-BERT: Adversarial Training BERT for Acronym Identification Winning Solution for SDU@AAAI-21cited by 7
13 more
  1. 2024SIKeD: Self-guided Iterative Knowledge Distillation for mathematical reasoningcited by 6
  2. 2024CleaR: Towards Robust and Generalized Parameter-Efficient Fine-Tuning for Noisy Label Learningcited by 6
  3. 2021How Emotionally Stable is ALBERT? Testing Robustness with Stochastic Weight Averaging on a Sentiment Analysis Taskcited by 5
  4. 2022Unified and Effective Ensemble Knowledge Distillationcited by 5
  5. 2024Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillationcited by 5
  6. 2024Preserving Pre-trained Representation Space: On Effectiveness of Prefix-tuning for Large Multi-modal Modelscited by 4
  7. 2022Ensemble Transformer for Efficient and Accurate Ranking Tasks: an Application to Question Answering Systemscited by 3
  8. 2022Reduce, Reuse, Recycle: Improving Training Efficiency with Distillationcited by 3
  9. 2022SEMI-FND: Stacked Ensemble Based Multimodal Inference For Faster Fake News Detectioncited by 2
  10. 2025MaxPoolBERT: Enhancing BERT Classification via Layer- and Token-Wise Aggregationcited by 2
  11. 2023SurreyAI 2023 Submission for the Quality Estimation Shared Taskcited by 1
  12. 2021An Automated Knowledge Mining and Document Classification System with Multi-model Transfer Learningcited by 0
  13. 2024KPC-cF: Aspect-Based Sentiment Analysis via Implicit-Feature Alignment with Corpus Filteringcited by 0