# Improving BERT Fine-Tuning via Self-Ensemble and Self-Distillation

arXiv: https://arxiv.org/abs/2002.10345 · HTML: https://mlsanity.com/paper/2002.10345/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Feb 2020.

## Built on by (most cited first, 25 of 25)

- [Speculative Decoding with Big Little Decoder](https://mlsanity.com/paper/2302.07863/) (2023)
- [BiBERT: Accurate Fully Binarized BERT](https://mlsanity.com/paper/2203.06390/) (2022)
- [Unified Implicit Neural Stylization](https://mlsanity.com/paper/2204.01943/) (2022)
- [Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Data](https://mlsanity.com/paper/2107.00051/) (2021)
- [Knowledge Distillation for Federated Learning: a Practical Guide](https://mlsanity.com/paper/2211.04742/) (2022)
- [MirrorWiC: On Eliciting Word-in-Context Representations from Pretrained Language Models](https://mlsanity.com/paper/2109.09237/) (2021)
- [Alternative Input Signals Ease Transfer in Multilingual Machine Translation](https://mlsanity.com/paper/2110.07804/) (2021)
- [Multi-CLS BERT: An Efficient Alternative to Traditional Ensembling](https://mlsanity.com/paper/2210.05043/) (2022)
- [Cluster & Tune: Boost Cold Start Performance in Text Classification](https://mlsanity.com/paper/2203.10581/) (2022)
- [Neighborhood Consensus Contrastive Learning for Backward-Compatible Representation](https://mlsanity.com/paper/2108.03372/) (2021)
- [Large Language Models for Relevance Judgment in Product Search](https://mlsanity.com/paper/2406.00247/) (2024)
- [AT-BERT: Adversarial Training BERT for Acronym Identification Winning Solution for SDU@AAAI-21](https://mlsanity.com/paper/2101.03700/) (2021)
- [SIKeD: Self-guided Iterative Knowledge Distillation for mathematical reasoning](https://mlsanity.com/paper/2410.18574/) (2024)
- [CleaR: Towards Robust and Generalized Parameter-Efficient Fine-Tuning for Noisy Label Learning](https://mlsanity.com/paper/2411.00873/) (2024)
- [How Emotionally Stable is ALBERT? Testing Robustness with Stochastic Weight Averaging on a Sentiment Analysis Task](https://mlsanity.com/paper/2111.09612/) (2021)
- [Unified and Effective Ensemble Knowledge Distillation](https://mlsanity.com/paper/2204.00548/) (2022)
- [Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillation](https://mlsanity.com/paper/2411.06448/) (2024)
- [Preserving Pre-trained Representation Space: On Effectiveness of Prefix-tuning for Large Multi-modal Models](https://mlsanity.com/paper/2411.00029/) (2024)
- [Ensemble Transformer for Efficient and Accurate Ranking Tasks: an Application to Question Answering Systems](https://mlsanity.com/paper/2201.05767/) (2022)
- [Reduce, Reuse, Recycle: Improving Training Efficiency with Distillation](https://mlsanity.com/paper/2211.00683/) (2022)
- [SEMI-FND: Stacked Ensemble Based Multimodal Inference For Faster Fake News Detection](https://mlsanity.com/paper/2205.08159/) (2022)
- [MaxPoolBERT: Enhancing BERT Classification via Layer- and Token-Wise Aggregation](https://mlsanity.com/paper/2505.15696/) (2025)
- [SurreyAI 2023 Submission for the Quality Estimation Shared Task](https://mlsanity.com/paper/2312.00525/) (2023)
- [An Automated Knowledge Mining and Document Classification System with Multi-model Transfer Learning](https://mlsanity.com/paper/2106.12744/) (2021)
- [KPC-cF: Aspect-Based Sentiment Analysis via Implicit-Feature Alignment with Corpus Filtering](https://mlsanity.com/paper/2407.00342/) (2024)

## Abstract (arXiv)

Fine-tuning pre-trained language models like BERT has become an effective way in NLP and yields state-of-the-art results on many downstream tasks. Recent studies on adapting BERT to new tasks mainly focus on modifying the model structure, re-designing the pre-train tasks, and leveraging external data and knowledge. The fine-tuning strategy itself has yet to be fully explored. In this paper, we improve the fine-tuning of BERT with two effective mechanisms: self-ensemble and self-distillation. The experiments on text classification and natural language inference tasks show our proposed methods can significantly improve the adaption of BERT without any external data or knowledge.