# Generative Embedding Benchmark: How Much Information Survives in a Dense Embedding?

arXiv: https://arxiv.org/abs/2608.06972 · HTML: https://mlsanity.com/paper/2608.06972/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 29 references that are in the map.
- First posted on arXiv 7 Aug 2026.

## Builds on (most cited first, 21 of 21)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models](https://mlsanity.com/paper/2305.07895/) (2023)
- [Adapting Language Models to Compress Contexts](https://mlsanity.com/paper/2305.14788/) (2023)
- [Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking](https://mlsanity.com/paper/2601.04720/) (2026)
- [VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks](https://mlsanity.com/paper/2410.05160/) (2024)
- [Text Embeddings Reveal (Almost) As Much As Text](https://mlsanity.com/paper/2310.06816/) (2023)
- [A Theory of Usable Information Under Computational Constraints](https://mlsanity.com/paper/2002.10689/) (2020)
- [xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token](https://mlsanity.com/paper/2405.13792/) (2024)
- [MMTEB: Massive Multilingual Text Embedding Benchmark](https://mlsanity.com/paper/2502.13595/) (2025)
- [VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents](https://mlsanity.com/paper/2507.04590/) (2025)
- [Text-Only Training for Image Captioning using Noise-Injected CLIP](https://mlsanity.com/paper/2211.00575/) (2022)
- [DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Training](https://mlsanity.com/paper/2303.03032/) (2023)
- [Sentence Embedding Leaks More Information than You Expect: Generative Embedding Inversion Attack to Recover the Whole Sentence](https://mlsanity.com/paper/2305.03010/) (2023)
- [Embed-RL: Reinforcement Learning for Reasoning-Driven Multimodal Embeddings](https://mlsanity.com/paper/2602.13823/) (2026)
- [Training LLMs to be Better Text Embedders through Bidirectional Reconstruction](https://mlsanity.com/paper/2509.03020/) (2025)
- [Implicit Inversion turns CLIP into a Decoder](https://mlsanity.com/paper/2505.23161/) (2025)

## Abstract (arXiv)

Embeddings have emerged as a standard representational interface linking foundation models with downstream systems. Most embedding benchmarks assess representations through discriminative tasks or geometric criteria centered on separability in embedding space. However, strong performance on such evaluations does not establish whether content compressed into an embedding remains accessible to a downstream generator. To address this gap, we introduce the Generative Embedding Benchmark (GEB), in which a decoder answers questions using only a frozen embedding and question text, without access to the original image or intermediate visual features. Answer quality under this readout measures generative information: the answer-relevant content recoverable from an embedding. GEB includes a curated visual-question-answering dataset with a 1,800-item development split and a held-out 900-item test split covering natural images, scene text, and visual documents. Using a common decoder and training recipe, we evaluate seven public embedding models in visual-only and vision-language joint modes. On the test set, visual-only scores range from 28.25 to 33.21; with image-question joint encoding, all five VLM-based embedding models score higher, and the best reaches 65.56. Matched embeddings also outperform text-only inputs, zero embeddings, and shuffled embeddings. Natural-image information is much easier to recover than scene text or visual-document information, while a Qwen3-VL-2B reference with access to the original image reaches 84.30. Together, these results show that generative readout exposes information bottlenecks that separability-based evaluation does not capture.