ANCHOR: LLM-driven Subject Conditioning for Text-to-Image Synthesis
Not on the map. Why?
Abstract (arXiv)
Text-to-image (T2I) models have achieved remarkable progress in high-quality image synthesis, yet most benchmarks rely on simple, self-contained prompts, failing to capture the complexity of real-world captions. Human-written captions often involve multiple interacting subjects, rich contextual references, and abstractive phrasing, conditions under which current image-text encoders like CLIP struggle. To systematically study these deficiencies, we introduce ANCHOR, a large-scale dataset of 70K+ abstractive captions sourced from five major news media organizations. Analysis with ANCHOR reveals persistent failures in multi-subject understanding, context reasoning, and nuanced grounding. Motivated by these challenges, we propose Subject-Aware Fine-tuning (SAFE), which uses Large Language Models (LLMs) to extract key subjects and enhance their representation at the embedding-level. Experiments with contemporary models show that SAFE significantly improves image-caption consistency and human preference alignment, serving as a practical and scalable solution.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 47 of its 79 references that are in the map. First posted on arXiv 15 Apr 2024.
Builds on
47 of 79 references.
References · 47
- 2017Attention Is All You Needcited by 59,932
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
- 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
- 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
- 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
- 2021Zero-Shot Text-to-Image Generationcited by 4,276
- 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
- 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
- 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
- 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
- 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
35 more
- 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
- 2016Understanding deep learning requires rethinking generalizationcited by 1,657
- 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
- 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
- 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
- 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
- 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
- 2022Exploring CLIP for Assessing the Look and Feel of Imagescited by 974
- 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
- 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
- 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
- 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
- 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 516
- 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Modelscited by 376
- 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
- 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
- 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
- 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 245
- 2022The Role of ImageNet Classes in Fr'echet Inception Distancecited by 211
- 2020Visual News: Benchmark and Challenges in News Image Captioningcited by 121
- 2022BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotationscited by 99
- 2023Inserting Anybody in Diffusion Models via Celeb Basiscited by 63
- 2020REL: An Entity Linker Standing on the Shoulders of Giantscited by 61
- 2020Transform and Tell: Entity-Aware News Image Captioningcited by 52
- 2024A Sober Look at the Robustness of CLIPs to Spurious Featurescited by 38
- 2024VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learningcited by 34
- 2024Facial Affective Behavior Analysis with Instruction Tuningcited by 26
- 2020Clue: Cross-modal Coherence Modeling for Caption Generationcited by 24
- 2023Text-to-Image Generation for Abstract Conceptscited by 18
- 2024Can MLLMs Perform Text-to-Image In-Context Learning?cited by 13
- 2019CITE: A Corpus of Image-Text Discourse Relationscited by 12
- 2021The Role of Syntactic Planning in Compositional Image Captioningcited by 12
- 2022Focus! Relevant and Sufficient Context Selection for News Image Captioningcited by 10
- 2023Understanding Social Media Cross-Modality Discourse in Linguistic Spacecited by 6
- 2021Exploring Explicit and Implicit Visual Relationships for Image Captioningcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Structured Captions Improve Prompt Adherence in Text-to-Image Models (Re-LAION-Caption 19M)
- 2024No Detail Left Behind: Revisiting Self-Retrieval for Fine-Grained Image Captioning
- 2025CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learning
- 2025Semantic Anchoring for Robust Personalization in Text-to-Image Diffusion Models
- 2026Squeezing Capacity from Multimodal Large Language Models for Subject-driven Generation
- 2025Free Lunch Alignment of Text-to-Image Diffusion Models without Preference Image Pairs
- 2023A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image Generation
- 2023CapsFusion: Rethinking Image-Text Data at Scale