# t-gems: text-guided exit modules for decreasing clip image encoder

arXiv: https://arxiv.org/abs/2605.17499 · HTML: https://mlsanity.com/paper/2605.17499/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 25 references that are in the map.
- First posted on arXiv 17 May 2026.

## Builds on (most cited first, 18 of 18)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Token Merging: Your ViT But Faster](https://mlsanity.com/paper/2210.09461/) (2022)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [AdaViT: Adaptive Tokens for Efficient Vision Transformer](https://mlsanity.com/paper/2112.07658/) (2021)
- [DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference](https://mlsanity.com/paper/2004.12993/) (2020)
- [Depth-Adaptive Transformer](https://mlsanity.com/paper/1910.10073/) (2019)
- [Consistent Accelerated Inference via Confident Adaptive Transformers](https://mlsanity.com/paper/2104.08803/) (2021)
- [CascadeBERT: Accelerating Inference of Pre-trained Language Models via Calibrated Complete Models Cascade](https://mlsanity.com/paper/2012.14682/) (2020)
- [You Need Multiple Exiting: Dynamic Early Exiting for Accelerating Unified Vision Language Model](https://mlsanity.com/paper/2211.11152/) (2022)
- [A Simple Hash-Based Early Exiting Approach For Language Understanding and Generation](https://mlsanity.com/paper/2203.01670/) (2022)
- [DACT-BERT: Differentiable Adaptive Computation Time for an Efficient BERT Inference](https://mlsanity.com/paper/2109.11745/) (2021)
- [Adaptive Deep Neural Network Inference Optimization with EENet](https://mlsanity.com/paper/2301.07099/) (2023)
- [Improving the Trainability of Deep Neural Networks through Layerwise Batch-Entropy Regularization](https://mlsanity.com/paper/2208.01134/) (2022)

## Abstract (arXiv)

Multimodal deep neural networks enhance deep comprehension by integrating diverse data modalities. Data from different modalities are typically projected into a shared latent space for similarity computation, but this process is resource intensive due to large image encoders and equal processing of test data during prediction. Early exit methods reduce computational load by utilizing intermediate layers, saving time and memory. However, developing such methods is challenging for multimodal data like image-text pairs. This study investigates the semantic content distributions present in intermediate layers of encoders such as CLIP, which can be derived from textual descriptions. We introduce Text-Guided Exit Modules (T-GEMs) and a rate-based regularizer to control encoder usage costs while maintaining cross-modal understanding performance.