# E-LANG: Energy-Based Joint Inferencing of Super and Swift Language Models

arXiv: https://arxiv.org/abs/2203.00748 · HTML: https://mlsanity.com/paper/2203.00748/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 40 references that are in the map.
- First posted on arXiv 1 Mar 2022.

## Builds on (most cited first, 30 of 32)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices](https://mlsanity.com/paper/2004.02984/) (2020)
- [Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT](https://mlsanity.com/paper/1909.05840/) (2019)
- [DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference](https://mlsanity.com/paper/2004.12993/) (2020)
- [A Survey of Model Compression and Acceleration for Deep Neural Networks](https://mlsanity.com/paper/1710.09282/) (2017)
- [Model compression via distillation and quantization](https://mlsanity.com/paper/1802.05668/) (2018)
- [The Lottery Ticket Hypothesis for Pre-trained BERT Networks](https://mlsanity.com/paper/2007.12223/) (2020)
- [I-BERT: Integer-only BERT Quantization](https://mlsanity.com/paper/2101.01321/) (2021)
- [DynaBERT: Dynamic BERT with Adaptive Width and Depth](https://mlsanity.com/paper/2004.04037/) (2020)
- [Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning](https://mlsanity.com/paper/2002.08307/) (2020)
- [BinaryBERT: Pushing the Limit of BERT Quantization](https://mlsanity.com/paper/2012.15701/) (2020)
- [An Energy and GPU-Computation Efficient Backbone Network for Real-Time Object Detection](https://mlsanity.com/paper/1904.09730/) (2019)
- [TernaryBERT: Distillation-aware Ultra-low Bit BERT](https://mlsanity.com/paper/2009.12812/) (2020)
- [Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search](https://mlsanity.com/paper/2010.07003/) (2020)
- [Compression of Deep Learning Models for Text: A Survey](https://mlsanity.com/paper/2008.05221/) (2020)
- [SqueezeBERT: What can computer vision teach NLP about efficient neural networks?](https://mlsanity.com/paper/2006.11316/) (2020)
- [Auto-Split: A General Framework of Collaborative Edge-Cloud AI](https://mlsanity.com/paper/2108.13041/) (2021)
- [ReasoNet: Learning to Stop Reading in Machine Comprehension](https://mlsanity.com/paper/1609.05284/) (2016)
- [KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size Quantization](https://mlsanity.com/paper/2101.05938/) (2021)

## Built on by (most cited first, 8 of 8)

- [ETran: Energy-Based Transferability Estimation](https://mlsanity.com/paper/2308.02027/) (2023)
- [DEED: Dynamic Early Exit on Decoder for Accelerating Encoder-Decoder Transformer Models](https://mlsanity.com/paper/2311.08623/) (2023)
- [GOLD: Generalized Knowledge Distillation via Out-of-Distribution-Guided Language Data Generation](https://mlsanity.com/paper/2403.19754/) (2024)
- [LEVI: Generalizable Fine-tuning via Layer-wise Ensemble of Different Views](https://mlsanity.com/paper/2402.04644/) (2024)
- [Extending Momentum Contrast with Cross Similarity Consistency Regularization](https://mlsanity.com/paper/2206.04676/) (2022)
- [Improving Prediction Certainty Estimation for Reliable Early Exiting via Null Space Projection](https://mlsanity.com/paper/2506.17249/) (2025)
- [ArchBERT: Bi-Modal Understanding of Neural Architectures and Natural Languages](https://mlsanity.com/paper/2310.17737/) (2023)
- [Spatial Cross-Attention Improves Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/2206.05028/) (2022)

## Abstract (arXiv)

Building huge and highly capable language models has been a trend in the past years. Despite their great performance, they incur high computational cost. A common solution is to apply model compression or choose light-weight architectures, which often need a separate fixed-size model for each desirable computational budget, and may lose performance in case of heavy compression. This paper proposes an effective dynamic inference approach, called E-LANG, which distributes the inference between large accurate Super-models and light-weight Swift models. To this end, a decision making module routes the inputs to Super or Swift models based on the energy characteristics of the representations in the latent space. This method is easily adoptable and architecture agnostic. As such, it can be applied to black-box pre-trained models without a need for architectural manipulations, reassembling of modules, or re-training. Unlike existing methods that are only applicable to encoder-only backbones and classification tasks, our method also works for encoder-decoder structures and sequence-to-sequence tasks such as translation. The E-LANG performance is verified through a set of experiments with T5 and BERT backbones on GLUE, SuperGLUE, and WMT. In particular, we outperform T5-11B with an average computations speed-up of 3.3× on GLUE and 2.9× on SuperGLUE. We also achieve BERT-based SOTA on GLUE with 3.2× less computations. Code and demo are available in the supplementary materials.