# Lost in Embeddings: Information Loss in Vision-Language Models

arXiv: https://arxiv.org/abs/2509.11986 · HTML: https://mlsanity.com/paper/2509.11986/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 13 of its 31 references that are in the map.
- First posted on arXiv 15 Sep 2025.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 13 of 13)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs](https://mlsanity.com/paper/2401.06209/) (2024)
- [Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning](https://mlsanity.com/paper/2203.02053/) (2022)
- [Why are Visually-Grounded Language Models Bad at Image Classification?](https://mlsanity.com/paper/2405.18415/) (2024)
- [Multimodal Alignment and Fusion: A Survey](https://mlsanity.com/paper/2411.17040/) (2024)
- [Connector-S: A Survey of Connectors in Multi-modal Large Language Models](https://mlsanity.com/paper/2502.11453/) (2025)

## Built on by (most cited first, 20 of 20)

- [Mull-Tokens: Modality-Agnostic Latent Thinking](https://mlsanity.com/paper/2512.10941/) (2025)
- [Large Vision-Language Models Get Lost in Attention](https://mlsanity.com/paper/2605.05668/) (2026)
- [SAMoE-VLA: A Scene Adaptive Mixture-of-Experts Vision-Language-Action Model for Autonomous Driving](https://mlsanity.com/paper/2603.08113/) (2026)
- [Vision-Language Models Unlock Task-Centric Latent Actions](https://mlsanity.com/paper/2601.22714/) (2026)
- [MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Image](https://mlsanity.com/paper/2605.10616/) (2026)
- [The Describe-Then-Generate Bottleneck: How VLM Descriptions Alter Image Generation Outcomes](https://mlsanity.com/paper/2509.18179/) (2025)
- [Text-Only Training for Image Captioning with Retrieval Augmentation and Modality Gap Correction](https://mlsanity.com/paper/2512.04309/) (2025)
- [Reasoning with Pixel-level Precision: QVLM Architecture and SQuID Dataset for Quantitative Geospatial Analytics](https://mlsanity.com/paper/2601.13401/) (2026)
- [AGMark: Attention-Guided Dynamic Watermarking for Large Vision-Language Models](https://mlsanity.com/paper/2602.09611/) (2026)
- [SimLBR: Learning to Detect Fake Images by Learning to Detect Real Images](https://mlsanity.com/paper/2602.20412/) (2026)
- [Statistical Embeddings for Similarity, Retrieval, and Interpretable Alignment of Numeric Tabular Datasets](https://mlsanity.com/paper/2605.30289/) (2026)
- [ConsensusDrop: Fusing Visual and Cross-Modal Saliency for Efficient Vision Language Models](https://mlsanity.com/paper/2602.00946/) (2026)
- [Reflection Anchors for Propagation-Aware Visual Retention in Long-Chain Multimodal Reasoning](https://mlsanity.com/paper/2605.09614/) (2026)
- [A Picture is Worth a Thousand Words? An Empirical Study of Aggregation Strategies for Visual Financial Document Retrieval](https://mlsanity.com/paper/2605.14581/) (2026)
- [V-LynX: Token Interface Alignment for Video+X LLMs](https://mlsanity.com/paper/2606.00508/) (2026)
- [The Ebb and Flow of Multimodal Focus: Scheduling Visual Relay Windows for Grounded VLM Reasoning](https://mlsanity.com/paper/2607.11436/) (2026)
- [Beyond Similarity: Foundation Models as an Efficient Backbone for Training-Free Composed Video Retrieval](https://mlsanity.com/paper/2609.10008/) (2026)
- [Certified Interface Aliases: Exact Collisions in Vision-Language Preprocessing, and When They Exist](https://mlsanity.com/paper/2609.33003/) (2026)
- [Matching Object or Relation? Tracing Abstract Reasoning Inside VLMs](https://mlsanity.com/paper/2610.07646/) (2026)
- [Rethinking Projector Training in Multimodal LLMs](https://mlsanity.com/paper/2610.18598/) (2026)

## Abstract (arXiv)

Vision--language models (VLMs) often process visual inputs through a pretrained vision encoder, followed by a projection into the language model's embedding space via a connector component. While crucial for modality fusion, the potential information loss induced by this projection step and its direct impact on model capabilities remain understudied. We introduce two complementary approaches to examine and quantify this loss by analyzing the latent representation space. First, we evaluate semantic information preservation by analyzing changes in k-nearest neighbor relationships between image representations, before and after projection. Second, we directly measure information loss by reconstructing visual embeddings from the projected representation, localizing loss at an image patch level. Experiments reveal that connectors substantially distort the local geometry of visual representations, with k-nearest neighbors diverging by 40--60% post-projection, correlating with degradation in retrieval performance. The patch-level embedding reconstruction provides interpretable insights for model behavior on visually grounded question-answering tasks, finding that areas of high information loss reliably predict instances where models struggle.