mlsanityData to 7 Oct 2026

Mutual Information Divergence: A Unified Metric for Multimodal Generative Models

arXiv 2205.13445 ↗First posted 25 May 2022cs.CV cs.AI cs.CLcited by 35

Not on the map. Why?

Abstract (arXiv)

Text-to-image generation and image captioning are recently emerged as a new experimental paradigm to assess machine intelligence. They predict continuous quantity accompanied by their sampling techniques in the generation, making evaluation complicated and intractable to get marginal distributions. Based on a recent trend that multimodal generative evaluations exploit a vison-and-language pre-trained model, we propose the negative Gaussian cross-mutual information using the CLIP features as a unified metric, coined by Mutual Information Divergence (MID). To validate, we extensively compare it with competing metrics using carefully-generated or human-annotated judgments in text-to-image generation and image captioning tasks. The proposed MID significantly outperforms the competitive methods by having consistency across benchmarks, sample parsimony, and robustness toward the exploited CLIP model. We look forward to seeing the underrepresented implications of the Gaussian cross-mutual information in multimodal representation learning and the future works based on this novel proposition.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 35 of its 62 references that are in the map. First posted on arXiv 25 May 2022.

Builds on

35 of 62 references.

References · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  6. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  7. 2016Improved Techniques for Training GANscited by 2,938
  8. 2018YOLOv3: An Incremental Improvementcited by 2,652
  9. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  11. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  12. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
23 more
  1. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 925
  2. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  3. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  4. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  5. 2018Object Hallucination in Image Captioningcited by 661
  6. 2020Reliable Fidelity and Diversity Metrics for Generative Modelscited by 425
  7. 2018Stacked Cross Attention for Image-Text Matchingcited by 414
  8. 2018Assessing Generative Models via Precision and Recallcited by 401
  9. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  10. 2016Learning Deep Representations of Fine-grained Visual Descriptionscited by 182
  11. 2018Inferring Semantic Layout for Hierarchical Text-to-Image Synthesiscited by 101
  12. 2017FOIL it! Find One mismatch between Image and Language captioncited by 95
  13. 2019Semantic Object Accuracy for Generative Text-to-Image Synthesiscited by 89
  14. 2018NSML: Meet the MLaaS platform with a real-world case studycited by 70
  15. 2019TIGEr: Text-to-Image Grounding for Image Caption Evaluationcited by 58
  16. 2018Learning to Evaluate Image Captioningcited by 53
  17. 2021Adversarial Text-to-Image Synthesis: A Reviewcited by 52
  18. 2022ECCV Caption: Correcting False Negatives by Collecting Machine-and-Human-verified Image-Caption Associations for MS-COCOcited by 37
  19. 2020NUBIA: NeUral Based Interchangeability Assessor for Text Generationcited by 34
  20. 2022Evidence for Hypodescent in Visual Semantic AIcited by 30
  21. 2020It's Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Informationcited by 26
  22. 2021Semi-orthogonal Embedding for Efficient Unsupervised Anomaly Segmentationcited by 26
  23. 2020Toward a Generalization Metric for Deep Generative Modelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 8; 2024: 16; 2025: 8; 2026: 2.

35 papers.

Sort:

By area:

  1. 2022FLAME: Free-form Language-based Motion Synthesis & Editingcited by 217
  2. 2023VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluationcited by 196
  3. 2023Human Motion Generation: A Surveycited by 95
  4. 2023LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluationcited by 85
  5. 2024Benchmarking and Improving Detail Image Captioncited by 81
  6. 2023Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluationcited by 78
  7. 2024Polos: Multimodal Metric Learning from Human Feedback for Image Captioningcited by 46
  8. 2024A Survey on Quality Metrics for Text-to-Image Generationcited by 17
  9. 2025Image Captioning Evaluation in the Age of Multimodal LLMs: Challenges and Future Perspectivescited by 17
  10. 2024Positive-Augmented Contrastive Learning for Vision-and-Language Evaluation and Trainingcited by 16
  11. 2024BRIDGE: Bridging Gaps in Image Captioning Evaluation with Stronger Visual Cuescited by 13
  12. 2024HICEScore: A Hierarchical Metric for Image Captioning Evaluationcited by 12
23 more
  1. 2024DENEB: A Hallucination-Robust Automatic Evaluation Metric for Image Captioningcited by 11
  2. 2024PAK-UCB Contextual Bandit: An Online Learning Approach to Prompt-Aware Selection of Generative Models and LLMscited by 11
  3. 2023Multi-modal Latent Diffusioncited by 9
  4. 20254D-Bench: Benchmarking Multi-modal Large Language Models for 4D Object Understandingcited by 8
  5. 2024When is an Embedding Model More Promising than Another?cited by 7
  6. 2025Distributional Vision-Language Alignment by Cauchy-Schwarz Divergencecited by 7
  7. 2023Text-Conditioned Sampling Framework for Text-to-Image Generation with Masked Generative Modelscited by 6
  8. 2024textttCOSMIC: Mutual Information for Task-Agnostic Summarization Evaluationcited by 6
  9. 2024Information-Theoretic Distillation for Reference-less Summarizationcited by 6
  10. 2024Conditional Vendi Score: Prompt-Aware Diversity Evaluation for Generative AI Models and LLMscited by 6
  11. 2023SelfEval: Leveraging the discriminative nature of generative models for evaluationcited by 5
  12. 2024Text-to-Image Cross-Modal Generation: A Systematic Reviewcited by 5
  13. 2024Toward Automatic Relevance Judgment using Vision--Language Models for Image--Text Retrieval Evaluationcited by 5
  14. 2025Evaluation of Multilingual Image Captioning: How far can we get with CLIP models?cited by 5
  15. 2025A Rate-Distortion Framework for Summarizationcited by 3
  16. 2025DISCODE: Distribution-Aware Score Decoder for Robust Automatic Evaluation of Image Captioningcited by 2
  17. 2026MI-Pruner: Crossmodal Mutual Information-guided Token Pruner for Efficient MLLMscited by 2
  18. 2024AgentPeerTalk: Empowering Students through Agentic-AI-Driven Discernment of Bullying and Joking in Peer Interactions in Schoolscited by 1
  19. 2023Linear Alignment of Vision-language Models for Image Captioningcited by 0
  20. 2024DRIVE: Dual-Robustness via Information Variability and Entropic Consistency in Source-Free Unsupervised Domain Adaptationcited by 0
  21. 2025A Conformal Risk Control Framework for Granular Word Assessment and Uncertainty Calibration of CLIPScore Quality Estimatescited by 0
  22. 2025Redemption Score: A Multi-Modal Evaluation Framework for Image Captioning via Distributional, Perceptual, and Linguistic Signal Triangulationcited by 0
  23. 2026ITIScore: An Image-to-Text-to-Image Rating Framework for the Image Captioning Ability of MLLMscited by 0