mlsanityData to 7 Oct 2026

ANCHOR: LLM-driven Subject Conditioning for Text-to-Image Synthesis

arXiv 2404.10141 ↗First posted 15 Apr 2024cs.CV cs.CL cs.MMcited by 0

Not on the map. Why?

Abstract (arXiv)

Text-to-image (T2I) models have achieved remarkable progress in high-quality image synthesis, yet most benchmarks rely on simple, self-contained prompts, failing to capture the complexity of real-world captions. Human-written captions often involve multiple interacting subjects, rich contextual references, and abstractive phrasing, conditions under which current image-text encoders like CLIP struggle. To systematically study these deficiencies, we introduce ANCHOR, a large-scale dataset of 70K+ abstractive captions sourced from five major news media organizations. Analysis with ANCHOR reveals persistent failures in multi-subject understanding, context reasoning, and nuanced grounding. Motivated by these challenges, we propose Subject-Aware Fine-tuning (SAFE), which uses Large Language Models (LLMs) to extract key subjects and enhance their representation at the embedding-level. Experiments with contemporary models show that SAFE significantly improves image-caption consistency and human preference alignment, serving as a practical and scalable solution.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 47 of its 79 references that are in the map. First posted on arXiv 15 Apr 2024.

Builds on

47 of 79 references.

References · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  9. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  10. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  11. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  12. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
35 more
  1. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  3. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  4. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  5. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  6. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  7. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  8. 2022Exploring CLIP for Assessing the Look and Feel of Imagescited by 974
  9. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  10. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  11. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  12. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  13. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 516
  14. 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Modelscited by 376
  15. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  16. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  17. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  18. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 245
  19. 2022The Role of ImageNet Classes in Fr'echet Inception Distancecited by 211
  20. 2020Visual News: Benchmark and Challenges in News Image Captioningcited by 121
  21. 2022BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotationscited by 99
  22. 2023Inserting Anybody in Diffusion Models via Celeb Basiscited by 63
  23. 2020REL: An Entity Linker Standing on the Shoulders of Giantscited by 61
  24. 2020Transform and Tell: Entity-Aware News Image Captioningcited by 52
  25. 2024A Sober Look at the Robustness of CLIPs to Spurious Featurescited by 38
  26. 2024VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learningcited by 34
  27. 2024Facial Affective Behavior Analysis with Instruction Tuningcited by 26
  28. 2020Clue: Cross-modal Coherence Modeling for Caption Generationcited by 24
  29. 2023Text-to-Image Generation for Abstract Conceptscited by 18
  30. 2024Can MLLMs Perform Text-to-Image In-Context Learning?cited by 13
  31. 2019CITE: A Corpus of Image-Text Discourse Relationscited by 12
  32. 2021The Role of Syntactic Planning in Compositional Image Captioningcited by 12
  33. 2022Focus! Relevant and Sufficient Context Selection for News Image Captioningcited by 10
  34. 2023Understanding Social Media Cross-Modality Discourse in Linguistic Spacecited by 6
  35. 2021Exploring Explicit and Implicit Visual Relationships for Image Captioningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Structured Captions Improve Prompt Adherence in Text-to-Image Models (Re-LAION-Caption 19M)
  2. 2024No Detail Left Behind: Revisiting Self-Retrieval for Fine-Grained Image Captioning
  3. 2025CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learning
  4. 2025Semantic Anchoring for Robust Personalization in Text-to-Image Diffusion Models
  5. 2026Squeezing Capacity from Multimodal Large Language Models for Subject-driven Generation
  6. 2025Free Lunch Alignment of Text-to-Image Diffusion Models without Preference Image Pairs
  7. 2023A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image Generation
  8. 2023CapsFusion: Rethinking Image-Text Data at Scale