mlsanityData to 7 Oct 2026

DACT-BERT: Differentiable Adaptive Computation Time for an Efficient BERT Inference

arXiv 2109.11745 ↗First posted 24 Sep 2021cs.CL cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

Large-scale pre-trained language models have shown remarkable results in diverse NLP applications. Unfortunately, these performance gains have been accompanied by a significant increase in computation time and model size, stressing the need to develop new or complementary strategies to increase the efficiency of these models. In this paper we propose DACT-BERT, a differentiable adaptive computation time strategy for BERT-like models. DACT-BERT adds an adaptive computational mechanism to BERT's regular processing pipeline, which controls the number of Transformer blocks that need to be executed at inference time. By doing this, the model learns to combine the most appropriate intermediate representations for the task at hand. Our experiments demonstrate that our approach, when compared to the baselines, excels on a reduced computational regime and is competitive in other less restrictive ones.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 20 of its 26 references that are in the map. First posted on arXiv 24 Sep 2021.

Builds on

20 of 26 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2017On Calibration of Modern Neural Networkscited by 4,860
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  7. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  8. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  9. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  10. 2019Green AIcited by 620
  11. 2018Universal Transformerscited by 613
  12. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
8 more
  1. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  2. 2021Dynamic Neural Networks: A Surveycited by 405
  3. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  4. 2019Revealing the Dark Secrets of BERTcited by 300
  5. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  6. 2019Depth-Adaptive Transformercited by 227
  7. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  8. 2020Differentiable Adaptive Computation Time for Visual Reasoningcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 3; 2024: 0; 2025: 1; 2026: 2.

9 papers.

  1. 2023Think before you speak: Training Language Models With Pause Tokenscited by 254
  2. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruningcited by 56
  3. 2022AdapLeR: Speeding up Inference by Adaptive Length Reductioncited by 19
  4. 2025Skip a Layer or Loop it? Test-Time Depth Adaptation of Pretrained LLMscited by 14
  5. 2022Entropy-based Stability-Plasticity for Lifelong Learningcited by 8
  6. 2026Skip a Layer or Loop It? Learning Program-of-Layers in LLMscited by 6
  7. 2023Efficient Encoders for Streaming Sequence Taggingcited by 4
  8. 2023Modular Transformers: Compressing Transformers into Modularized Layers for Flexible Efficient Inferencecited by 4
  9. 2026t-gems: text-guided exit modules for decreasing clip image encodercited by 0