mlsanityData to 13 Oct 2026

Efficient Adaptive Transformer: An Empirical Study and Reproducible Framework

arXiv 2510.12856 ↗First posted 14 Oct 2025cs.CL cs.AIcited by 0

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

The Efficient Adaptive Transformer (EAT) framework unifies three adaptive efficiency techniques - progressive token pruning, sparse attention, and dynamic early exiting - into a single, reproducible architecture for input-adaptive inference. EAT provides an open-source benchmarking pipeline that automates data processing, timing, and ablation across GLUE tasks (SST-2, QQP, MNLI). Although this empirical study finds that combining these mechanisms can increase latency in shallow six-layer models, it demonstrates that EAT achieves slightly higher accuracy than the optimized DistilBERT baseline on SST-2, illustrating the potential of dynamic computation for latency-sensitive NLP. The main contribution is the open, end-to-end reproducible framework - complete with scripts, CSV logging, and analysis utilities - intended to serve as a community tool for further research on adaptive transformers.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 17 of its 21 references that are in the map. First posted on arXiv 14 Oct 2025.

Builds on

17 of 21 references.

References · 17

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,180
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,281
  5. 2020Big Bird: Transformers for Longer Sequencescited by 1,681
  6. 2020Reformer: The Efficient Transformercited by 1,662
  7. 2020Rethinking Attention with Performerscited by 1,594
  8. 2020Linformer: Self-Attention with Linear Complexitycited by 1,469
  9. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,157
  10. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 592
  11. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 436
  12. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 326
5 more
  1. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 270
  2. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 247
  3. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  4. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 184
  5. 2021Learned Token Pruning for Transformerscited by 139

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025On-the-Fly Adaptive Distillation of Transformer to Dual-State Linear Attention
  2. 2026ADEPT: Adaptive Dynamic Early-Exit Process for Transformers
  3. 2024A Survey of Transformer-based Language Models with Focus on Efficiency
  4. 2025LLMCache: Layer-Wise Caching Strategies for Accelerated Reuse in Transformer Inference
  5. 2025Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers
  6. 2026Adaptive Loops and Memory in Transformers: Think Harder or Know More?
  7. 2025Pimba: A Processing-in-Memory Acceleration for Post-Transformer Large Language Model Serving
  8. 2024Do Efficient Transformers Really Save Computation?