mlsanityData to 7 Oct 2026

ESsEN: Training Compact Discriminative Vision-Language Transformers in a Low-Resource Setting

arXiv 2604.18452 ↗First posted 20 Apr 2026cs.CV cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language modeling is rapidly increasing in popularity with an ever expanding list of available models. In most cases, these vision-language models have parameters in the tens of billions, which is necessary for some needs, but in many cases smaller models are necessary (e.g., on edge devices or independent robotic platforms). Unfortunately, there is little research in producing light-weight models or in training them with small datasets. Inspired by the language learning progression and data sparsity in child development, in this paper, we address both of these goals in a systematic fashion. We show that two-tower encoder models are superior to one-tower encoders in low-resource settings for discriminative English tasks. We show also that incorporating traditional convolutional networks into the two-tower transformer architecture can help produce parameter efficient vision-language models. Finally, we show that the cross-modal fusion module of two-tower encoders can vary significantly in shape and size while producing the same results. In addition, we present ESsEN, a compact vision-language model that can be trained end-to-end with relatively few resources that performs as well on several tasks with only a fraction of the parameters compared to other models. The experimental results and the tools we present here make vision-language modeling more accessible to a wider variety of researchers.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 33 references that are in the map. First posted on arXiv 20 Apr 2026.

Builds on

20 of 33 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  9. 2015VQA: Visual Question Answeringcited by 2,489
  10. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  11. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  12. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
8 more
  1. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  2. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruningcited by 56
  3. 2020MiniVLM: A Smaller and Faster Vision-Language Modelcited by 51
  4. 2024Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 48
  5. 2015Resolving References to Objects in Photographs using the Words-As-Classifiers Modelcited by 9
  6. 2024Renaissance: Investigating the Pretraining of Vision-Language Encoderscited by 2
  7. 2025Looking to Learn: Token-wise Dynamic Gating for Low-Resource Vision-Language Modellingcited by 2
  8. 2025What is the Best Sequence Length for BABYLM?cited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025NanoVLMs: How small can we go and still make coherent Vision Language Models?
  2. 2024SOLO: A Single Transformer for Scalable Vision-Language Modeling
  3. 2022Uni-EDEN: Universal Encoder-Decoder Network by Multi-Granular Vision-Language Pre-training
  4. 2025Perception Encoder: The best visual embeddings are not at the output of the network
  5. 2025EVEv2: Improved Baselines for Encoder-Free Vision-Language Models
  6. 2026PRISM-VLM: A Multi-Axis Discriminative Benchmark for Compact Vision-Language Models
  7. 2024Unveiling Encoder-Free Vision-Language Models
  8. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Models