# F-PABEE: Flexible-patience-based Early Exiting for Single-label and Multi-label text Classification Tasks

arXiv: https://arxiv.org/abs/2305.11916 · HTML: https://mlsanity.com/paper/2305.11916/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 44 references that are in the map.
- First posted on arXiv 21 May 2023.

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [Universal Transformers](https://mlsanity.com/paper/1807.03819/) (2018)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [Patient Knowledge Distillation for BERT Model Compression](https://mlsanity.com/paper/1908.09355/) (2019)
- [Reducing Transformer Depth on Demand with Structured Dropout](https://mlsanity.com/paper/1909.11556/) (2019)
- [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference](https://mlsanity.com/paper/2004.12993/) (2020)
- [BERT Loses Patience: Fast and Robust Inference with Early Exit](https://mlsanity.com/paper/2006.04152/) (2020)
- [I-BERT: Integer-only BERT Quantization](https://mlsanity.com/paper/2101.01321/) (2021)
- [FastBERT: a Self-distilling BERT with Adaptive Inference Time](https://mlsanity.com/paper/2004.02178/) (2020)
- [BinaryBERT: Pushing the Limit of BERT Quantization](https://mlsanity.com/paper/2012.15701/) (2020)
- [The Right Tool for the Job: Matching Model and Instance Complexities](https://mlsanity.com/paper/2004.07453/) (2020)
- [BERT-of-Theseus: Compressing BERT by Progressive Module Replacing](https://mlsanity.com/paper/2002.02925/) (2020)
- [SGM: Sequence Generation Model for Multi-label Classification](https://mlsanity.com/paper/1806.04822/) (2018)
- [Consistent Accelerated Inference via Confident Adaptive Transformers](https://mlsanity.com/paper/2104.08803/) (2021)
- [A Survey on Dynamic Neural Networks for Natural Language Processing](https://mlsanity.com/paper/2202.07101/) (2022)
- [Continuous Detection, Rapidly React: Unseen Rumors Detection based on Continual Prompt-Tuning](https://mlsanity.com/paper/2203.11720/) (2022)
- [Shallow-Deep Networks: Understanding and Mitigating Network Overthinking](https://mlsanity.com/paper/1810.07052/) (2018)
- [MVP-BERT: Redesigning Vocabularies for Chinese BERT and Multi-Vocab Pretraining](https://mlsanity.com/paper/2011.08539/) (2020)

## Built on by (most cited first, 18 of 18)

- [PromptCBLUE: A Chinese Prompt Tuning Benchmark for the Medical Domain](https://mlsanity.com/paper/2310.14151/) (2023)
- [ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Models](https://mlsanity.com/paper/2403.16187/) (2024)
- [MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning](https://mlsanity.com/paper/2410.18035/) (2024)
- [AMAS: Adaptively Determining Communication Topology for LLM-based Multi-Agent System](https://mlsanity.com/paper/2510.01617/) (2025)
- [PARA: Parameter-Efficient Fine-tuning with Prompt Aware Representation Adjustment](https://mlsanity.com/paper/2502.01033/) (2025)
- [FT-MDT: Extracting Decision Trees from Medical Texts via a Novel Low-rank Adaptation Method](https://mlsanity.com/paper/2510.04655/) (2025)
- [ADEPT: Adaptive Dynamic Early-Exit Process for Transformers](https://mlsanity.com/paper/2601.03700/) (2026)
- [MRAG: Benchmarking Retrieval-Augmented Generation for Bio-medicine](https://mlsanity.com/paper/2601.16503/) (2026)
- [Improving Prediction Certainty Estimation for Reliable Early Exiting via Null Space Projection](https://mlsanity.com/paper/2506.17249/) (2025)
- [Improving Prompt Tuning with Learned Prompting Layers](https://mlsanity.com/paper/2310.20127/) (2023)
- [IAPT: Instruction-Aware Prompt Tuning for Large Language Models](https://mlsanity.com/paper/2405.18203/) (2024)
- [COSEE: Consistency-Oriented Signal-Based Early Exiting via Calibrated Sample Weighting Mechanism](https://mlsanity.com/paper/2412.13236/) (2024)
- [EPEE: Towards Efficient and Effective Foundation Models in Biomedicine](https://mlsanity.com/paper/2503.02053/) (2025)
- [DeeAD: Dynamic Early Exit of Vision-Language Action for Efficient Autonomous Driving](https://mlsanity.com/paper/2511.20720/) (2025)
- [Evaluating ChatGPT on Medical Information Extraction Tasks: Performance, Explainability and Beyond](https://mlsanity.com/paper/2601.21767/) (2026)
- [Pursuing Best Industrial Practices for Retrieval-Augmented Generation in the Medical Domain](https://mlsanity.com/paper/2602.03368/) (2026)
- [ACL: Aligned Contrastive Learning Improves BERT and Multi-exit BERT Fine-tuning](https://mlsanity.com/paper/2602.03563/) (2026)
- [A Generative Model for Joint Multiple Intent Detection and Slot Filling](https://mlsanity.com/paper/2602.08322/) (2026)

## Abstract (arXiv)

Computational complexity and overthinking problems have become the bottlenecks for pre-training language models (PLMs) with millions or even trillions of parameters. A Flexible-Patience-Based Early Exiting method (F-PABEE) has been proposed to alleviate the problems mentioned above for single-label classification (SLC) and multi-label classification (MLC) tasks. F-PABEE makes predictions at the classifier and will exit early if predicted distributions of cross-layer are consecutively similar. It is more flexible than the previous state-of-the-art (SOTA) early exiting method PABEE because it can simultaneously adjust the similarity score thresholds and the patience parameters. Extensive experiments show that: (1) F-PABEE makes a better speedup-accuracy balance than existing early exiting strategies on both SLC and MLC tasks. (2) F-PABEE achieves faster inference and better performances on different PLMs such as BERT and ALBERT. (3) F-PABEE-JSKD performs best for F-PABEE with different similarity measures.