mlsanityData to 14 Oct 2026

What You See is What You Read? Improving Text-Image Alignment Evaluation

arXiv 2305.10400 ↗First posted 17 May 2023cs.CL cs.CVcited by 108

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Automatically determining whether a text and a corresponding image are semantically aligned is a significant challenge for vision-language models, with applications in generative text-to-image and image-to-text tasks. In this work, we study methods for automatic text-image alignment evaluation. We first introduce SeeTRUE: a comprehensive evaluation set, spanning multiple datasets from both text-to-image and image-to-text generation tasks, with human judgements for whether a given text-image pair is semantically aligned. We then describe two automatic methods to determine alignment: the first involving a pipeline based on question generation and visual question answering models, and the second employing an end-to-end classification approach by finetuning multimodal pretrained models. Both methods surpass prior approaches in various text-image alignment tasks, with significant improvements in challenging cases that involve complex composition or unnatural images. Finally, we demonstrate how our approaches can localize specific misalignments between an image and a given text, and how they can be used to automatically re-rank candidates in text-to-image generation.

Key facts. Cited by 108 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 52 of its 64 references that are in the map. First posted on arXiv 17 May 2023.

Builds on

52 of 64 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Cross-modal Attention Congruence Regularization for Vision-Language Relation Alignment20 Dec

Other references · 51

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  12. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
38 more
  1. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  2. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  3. 2015VQA: Visual Question Answeringcited by 2,506
  4. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  5. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  6. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  7. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  8. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  9. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  10. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 702
  11. 2018Object Hallucination in Image Captioningcited by 669
  12. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  13. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  14. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 428
  15. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 407
  16. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 378
  17. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 252
  18. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 244
  19. 2022Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpaintingcited by 233
  20. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  21. 2021Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answeringcited by 132
  22. 2022Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectorscited by 104
  23. 2022Benchmarking Spatial Relationships in Text-to-Image Generationcited by 97
  24. 2022A very preliminary analysis of DALL-E 2cited by 82
  25. 2023Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Imagescited by 80
  26. 2021Increasing Faithfulness in Knowledge-Grounded Dialogue with Controllable Featurescited by 74
  27. 2021Learning to Compose Visual Relationscited by 72
  28. 2022All You May Need for VQA are Image Captionscited by 65
  29. 2022Generated Faces in the Wild: Quantitative Comparison of Stable Diffusion, Midjourney and DALL-E 2cited by 55
  30. 2022DALLE-2 is Seeing Double: Flaws in Word-to-Concept Mapping in Text2Image Modelscited by 51
  31. 2022mFACE: Multilingual Summarization with Factual Consistency Evaluationcited by 42
  32. 2021Transparent Human Evaluation for Image Captioningcited by 40
  33. 2020Linguistically-Informed Transformations (LIT): A Method for Automatically Generating Contrast Setscited by 31
  34. 2021Automatic Generation of Contrast Sets from Scene Graphs: Probing the Compositional Consistency of GQAcited by 30
  35. 2021Learning Relation Alignment for Calibrated Cross-modal Retrievalcited by 21
  36. 2020Exposing Shallow Heuristics of Relation Extraction Models with Challenge Datacited by 20
  37. 2022Evaluating a Synthetic Image Dataset Generated with Stable Diffusioncited by 9
  38. 2022Do DALL-E and Flamingo Understand Each Other?cited by 7

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 14; 2024: 34; 2025: 40; 2026: 20.

108 papers.

Sort:

By area:

  1. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 474
  2. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 172
  3. 2023Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generationcited by 171
  4. 2025VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generationcited by 149
  5. 2024GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generationcited by 135
  6. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 132
  7. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 105
  8. 2024Improving Text-to-Image Consistency via Automatic Prompt Optimizationcited by 90
  9. 2024DreamMatcher: Appearance Matching Self-Attention for Semantically-Consistent Text-to-Image Personalizationcited by 60
  10. 2023DreamSync: Aligning Text-to-Image Generation with Image Understanding Feedbackcited by 58
  11. 2024ImageInWords: Unlocking Hyper-Detailed Image Descriptionscited by 57
  12. 2025Video-Bench: Human-Aligned Video Generation Benchmarkcited by 57
38 more
  1. 2023Revisiting the Role of Language Priors in Vision-Language Modelscited by 50
  2. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 47
  3. 2024Revisiting Text-to-Image Evaluation with Gecko: On Metrics, Prompts, and Human Ratingscited by 44
  4. 2023CLAIR: Evaluating Image Captions with Large Language Modelscited by 43
  5. 2024CounterCurate: Enhancing Physical and Semantic Visio-Linguistic Compositional Reasoning via Counterfactual Examplescited by 35
  6. 2025Easier Painting Than Thinking: Can Text-to-Image Models Set the Stage, but Not Direct the Play?cited by 34
  7. 2023Feedback-guided Data Synthesis for Imbalanced Classificationcited by 33
  8. 2023The Role of Chain-of-Thought in Complex Vision-Language Reasoning Taskcited by 32
  9. 2023VideoCon: Robust Video-Language Alignment via Contrast Captionscited by 30
  10. 2025NarrLV: Towards a Comprehensive Narrative-Centric Evaluation for Long Video Generationcited by 22
  11. 2024Holistic Unlearning Benchmark: A Multi-Faceted Evaluation for Text-to-Image Diffusion Model Unlearningcited by 21
  12. 2024FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehensioncited by 20
  13. 2025Maestro: Self-Improving Text-to-Image Generation via Agent Orchestrationcited by 19
  14. 2023Mismatch Quest: Visual and Textual Feedback for Image-Text Misalignmentcited by 18
  15. 2024A Survey on Quality Metrics for Text-to-Image Generationcited by 18
  16. 2024FINEMATCH: Aspect-based Fine-grained Image and Text Mismatch Detection and Correctioncited by 17
  17. 2025MMIG-Bench: Towards Comprehensive and Explainable Evaluation of Multi-Modal Image Generation Modelscited by 17
  18. 2024Evaluating Image Hallucination in Text-to-Image Generation with Question-Answeringcited by 16
  19. 2025Eval3D: Interpretable and Fine-grained Evaluation for 3D Generationcited by 16
  20. 2025REALEDIT: Reddit Edits As a Large-scale Empirical Dataset for Image Transformationscited by 15
  21. 2025CulturalFrames: Assessing Cultural Expectation Alignment in Text-to-Image Models and Evaluation Metricscited by 15
  22. 2023Discffusion: Discriminative Diffusion Models as Few-shot Vision and Language Learnerscited by 14
  23. 2024Automatic Evaluation for Text-to-image Generation: Task-decomposed Framework, Distilled Training, and Meta-evaluation Benchmarkcited by 14
  24. 2023Evaluation and Enhancement of Semantic Grounding in Large Vision-Language Modelscited by 13
  25. 2025Dynamic Classifier-Free Diffusion Guidance via Online Feedbackcited by 13
  26. 2024Looks Too Good To Be True: An Information-Theoretic Analysis of Hallucinations in Generative Restoration Modelscited by 12
  27. 2024Evaluating Numerical Reasoning in Text-to-Image Modelscited by 12
  28. 2024Exploring How Generative MLLMs Perceive More Than CLIP with the Same Vision Encodercited by 11
  29. 2024Towards Understanding and Quantifying Uncertainty for Text-to-Image Generationcited by 11
  30. 2024Visual Riddles: a Commonsense and World Knowledge Challenge for Large Vision and Language Modelscited by 10
  31. 2025On the Value of Cross-Modal Misalignment in Multimodal Representation Learningcited by 10
  32. 2025Fooling the LVLM Judges: Visual Biases in LVLM-Based Evaluationcited by 10
  33. 2025A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generationcited by 10
  34. 2023Visual Data-Type Understanding does not emerge from Scaling Vision-Language Modelscited by 9
  35. 2024EvolveDirector: Approaching Advanced Text-to-Image Generation with Large Vision-Language Modelscited by 9
  36. 2025Focus-N-Fix: Region-Aware Fine-Tuning for Text-to-Image Generationcited by 9
  37. 2024Removing Distributional Discrepancies in Captions Improves Image-Text Alignmentcited by 8
  38. 2024NewTerm: Benchmarking Real-Time New Terms for Large Language Models with Annual Updatescited by 8

50 most cited of 108 All 108 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Improving Compositional Text-to-image Generation with Large Vision-Language Models
  2. 2026Representations of Text and Images Align From Layer One
  3. 2025See the Text: From Tokenization to Visual Reading
  4. 2023Filter & Align: Leveraging Human Knowledge to Curate Image-Text Data
  5. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment
  6. 2023Divide, Evaluate, and Refine: Evaluating and Improving Text-to-Image Alignment with Iterative VQA Feedback
  7. 2024EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluation
  8. 2025Cycle Consistency as Reward: Learning Image-Text Alignment without Human Preferences