# Mutual Information Divergence: A Unified Metric for Multimodal Generative Models

arXiv: https://arxiv.org/abs/2205.13445 · HTML: https://mlsanity.com/paper/2205.13445/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 62 references that are in the map.
- First posted on arXiv 25 May 2022.

## Builds on (most cited first, 30 of 35)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [Improved Precision and Recall Metric for Assessing Generative Models](https://mlsanity.com/paper/1904.06991/) (2019)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [Vector Quantized Diffusion Model for Text-to-Image Synthesis](https://mlsanity.com/paper/2111.14822/) (2021)
- [Object Hallucination in Image Captioning](https://mlsanity.com/paper/1809.02156/) (2018)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)
- [Reliable Fidelity and Diversity Metrics for Generative Models](https://mlsanity.com/paper/2002.09797/) (2020)
- [Stacked Cross Attention for Image-Text Matching](https://mlsanity.com/paper/1803.08024/) (2018)
- [Assessing Generative Models via Precision and Recall](https://mlsanity.com/paper/1806.00035/) (2018)
- [DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesis](https://mlsanity.com/paper/1904.01310/) (2019)
- [Learning Deep Representations of Fine-grained Visual Descriptions](https://mlsanity.com/paper/1605.05395/) (2016)
- [Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis](https://mlsanity.com/paper/1801.05091/) (2018)
- [FOIL it! Find One mismatch between Image and Language caption](https://mlsanity.com/paper/1705.01359/) (2017)
- [Semantic Object Accuracy for Generative Text-to-Image Synthesis](https://mlsanity.com/paper/1910.13321/) (2019)
- [NSML: Meet the MLaaS platform with a real-world case study](https://mlsanity.com/paper/1810.09957/) (2018)
- [TIGEr: Text-to-Image Grounding for Image Caption Evaluation](https://mlsanity.com/paper/1909.02050/) (2019)
- [Learning to Evaluate Image Captioning](https://mlsanity.com/paper/1806.06422/) (2018)
- [Adversarial Text-to-Image Synthesis: A Review](https://mlsanity.com/paper/2101.09983/) (2021)
- [ECCV Caption: Correcting False Negatives by Collecting Machine-and-Human-verified Image-Caption Associations for MS-COCO](https://mlsanity.com/paper/2204.03359/) (2022)

## Built on by (most cited first, 30 of 35)

- [FLAME: Free-form Language-based Motion Synthesis & Editing](https://mlsanity.com/paper/2209.00349/) (2022)
- [VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation](https://mlsanity.com/paper/2312.14867/) (2023)
- [Human Motion Generation: A Survey](https://mlsanity.com/paper/2307.10894/) (2023)
- [LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluation](https://mlsanity.com/paper/2305.11116/) (2023)
- [Benchmarking and Improving Detail Image Caption](https://mlsanity.com/paper/2405.19092/) (2024)
- [Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluation](https://mlsanity.com/paper/2303.12112/) (2023)
- [Polos: Multimodal Metric Learning from Human Feedback for Image Captioning](https://mlsanity.com/paper/2402.18091/) (2024)
- [A Survey on Quality Metrics for Text-to-Image Generation](https://mlsanity.com/paper/2403.11821/) (2024)
- [Image Captioning Evaluation in the Age of Multimodal LLMs: Challenges and Future Perspectives](https://mlsanity.com/paper/2503.14604/) (2025)
- [Positive-Augmented Contrastive Learning for Vision-and-Language Evaluation and Training](https://mlsanity.com/paper/2410.07336/) (2024)
- [BRIDGE: Bridging Gaps in Image Captioning Evaluation with Stronger Visual Cues](https://mlsanity.com/paper/2407.20341/) (2024)
- [HICEScore: A Hierarchical Metric for Image Captioning Evaluation](https://mlsanity.com/paper/2407.18589/) (2024)
- [PAK-UCB Contextual Bandit: An Online Learning Approach to Prompt-Aware Selection of Generative Models and LLMs](https://mlsanity.com/paper/2410.13287/) (2024)
- [DENEB: A Hallucination-Robust Automatic Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2409.19255/) (2024)
- [Multi-modal Latent Diffusion](https://mlsanity.com/paper/2306.04445/) (2023)
- [4D-Bench: Benchmarking Multi-modal Large Language Models for 4D Object Understanding](https://mlsanity.com/paper/2503.17827/) (2025)
- [When is an Embedding Model More Promising than Another?](https://mlsanity.com/paper/2406.07640/) (2024)
- [Distributional Vision-Language Alignment by Cauchy-Schwarz Divergence](https://mlsanity.com/paper/2502.17028/) (2025)
- [Text-Conditioned Sampling Framework for Text-to-Image Generation with Masked Generative Models](https://mlsanity.com/paper/2304.01515/) (2023)
- [textttCOSMIC: Mutual Information for Task-Agnostic Summarization Evaluation](https://mlsanity.com/paper/2402.19457/) (2024)
- [Information-Theoretic Distillation for Reference-less Summarization](https://mlsanity.com/paper/2403.13780/) (2024)
- [Conditional Vendi Score: Prompt-Aware Diversity Evaluation for Generative AI Models and LLMs](https://mlsanity.com/paper/2411.02817/) (2024)
- [SelfEval: Leveraging the discriminative nature of generative models for evaluation](https://mlsanity.com/paper/2311.10708/) (2023)
- [Text-to-Image Cross-Modal Generation: A Systematic Review](https://mlsanity.com/paper/2401.11631/) (2024)
- [Toward Automatic Relevance Judgment using Vision--Language Models for Image--Text Retrieval Evaluation](https://mlsanity.com/paper/2408.01363/) (2024)
- [Evaluation of Multilingual Image Captioning: How far can we get with CLIP models?](https://mlsanity.com/paper/2502.06600/) (2025)
- [A Rate-Distortion Framework for Summarization](https://mlsanity.com/paper/2501.13100/) (2025)
- [DISCODE: Distribution-Aware Score Decoder for Robust Automatic Evaluation of Image Captioning](https://mlsanity.com/paper/2512.14420/) (2025)
- [MI-Pruner: Crossmodal Mutual Information-guided Token Pruner for Efficient MLLMs](https://mlsanity.com/paper/2604.03072/) (2026)
- [AgentPeerTalk: Empowering Students through Agentic-AI-Driven Discernment of Bullying and Joking in Peer Interactions in Schools](https://mlsanity.com/paper/2408.01459/) (2024)

## Abstract (arXiv)

Text-to-image generation and image captioning are recently emerged as a new experimental paradigm to assess machine intelligence. They predict continuous quantity accompanied by their sampling techniques in the generation, making evaluation complicated and intractable to get marginal distributions. Based on a recent trend that multimodal generative evaluations exploit a vison-and-language pre-trained model, we propose the negative Gaussian cross-mutual information using the CLIP features as a unified metric, coined by Mutual Information Divergence (MID). To validate, we extensively compare it with competing metrics using carefully-generated or human-annotated judgments in text-to-image generation and image captioning tasks. The proposed MID significantly outperforms the competitive methods by having consistency across benchmarks, sample parsimony, and robustness toward the exploited CLIP model. We look forward to seeing the underrepresented implications of the Gaussian cross-mutual information in multimodal representation learning and the future works based on this novel proposition.