mlsanityData to 14 Oct 2026

t-gems: text-guided exit modules for decreasing clip image encoder

arXiv 2605.17499 ↗First posted 17 May 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Multimodal deep neural networks enhance deep comprehension by integrating diverse data modalities. Data from different modalities are typically projected into a shared latent space for similarity computation, but this process is resource intensive due to large image encoders and equal processing of test data during prediction. Early exit methods reduce computational load by utilizing intermediate layers, saving time and memory. However, developing such methods is challenging for multimodal data like image-text pairs. This study investigates the semantic content distributions present in intermediate layers of encoders such as CLIP, which can be derived from textual descriptions. We introduce Text-Guided Exit Modules (T-GEMs) and a rate-based regularizer to control encoder usage costs while maintaining cross-modal understanding performance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 25 references that are in the map. First posted on arXiv 17 May 2026.

Builds on

18 of 25 references.

References · 18

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  4. 2022Token Merging: Your ViT But Fastercited by 1,021
  5. 2018Variational image compression with a scale hyperpriorcited by 935
  6. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  7. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 792
  8. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 596
  9. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 362
  10. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 332
  11. 2019Depth-Adaptive Transformercited by 236
  12. 2021Consistent Accelerated Inference via Confident Adaptive Transformerscited by 74
6 more
  1. 2020CascadeBERT: Accelerating Inference of Pre-trained Language Models via Calibrated Complete Models Cascadecited by 47
  2. 2022You Need Multiple Exiting: Dynamic Early Exiting for Accelerating Unified Vision Language Modelcited by 43
  3. 2022A Simple Hash-Based Early Exiting Approach For Language Understanding and Generationcited by 32
  4. 2021DACT-BERT: Differentiable Adaptive Computation Time for an Efficient BERT Inferencecited by 9
  5. 2023Adaptive Deep Neural Network Inference Optimization with EENetcited by 9
  6. 2022Improving the Trainability of Deep Neural Networks through Layerwise Batch-Entropy Regularizationcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Sparse CLIP: Co-Optimizing Interpretability and Performance in Contrastive Learning
  2. 2024Browse and Concentrate: Comprehending Multimodal Content via prior-LLM Context Fusion
  3. 2024Probing Multimodal Large Language Models for Global and Local Semantic Representations
  4. 2025BREEN: Bridge Data-Efficient Encoder-Free Multimodal Learning with Learnable Queries
  5. 2025Towards Multimodal Understanding via Stable Diffusion as a Task-Aware Feature Extractor
  6. 2025DeepInsert: Early Layer Bypass for Efficient and Performant Multimodal Understanding
  7. 2025HQ-CLIP: Leveraging Large Vision-Language Models to Create High-Quality Image-Text Datasets and CLIP Models
  8. 2024CLIP-CID: Efficient CLIP Distillation via Cluster-Instance Discrimination