The Plan, Not the Decoder: Diagnosing and Repairing Compositional Failure in Reasoning-Augmented Text-to-Image Generation
Not on the map. Why?
Abstract (arXiv)
Reasoning-augmented text-to-image models such as GoT-R1 emit an explicit textual plan - object names, attributes, and bounding boxes - before generating image tokens. When such a model fails a compositional prompt, is the plan wrong, or is the plan right and the decoder unfaithful? Because the plan is machine-readable it can be edited before decoding, which makes the two separable. We first validate the ruler. Swapping the two bounding boxes inside the model's own chain demonstrably flips the generated layout: detector-based accuracy falls 0.75 -> 0.48 (p<1e-3), while a widely used VQA-based spatial metric rises. A five-rater human study agrees with the detector on 81% of items and with the VQA judge on 57%. All spatial results therefore use geometric scoring. Under sound measurement the decoder is a faithful executor: 94% of generated layouts realize the planned relation, and object-box binding survives reordering of the plan's object segments. The planner is the bottleneck. It writes wrong relations for phrasing-dependent reasons - 98% accuracy on "left" against 54% on "right" for semantically identical layouts, a raster-order bias we isolate with a mention-order control - and cluttered geometry that the decoder faithfully reproduces. Editing the plan therefore fixes the image without retraining: symbolic verification with resampling gives +5.0 points (p<1e-3), minimal in-place repair +6.0 (p=.02), rewriting only box geometry +10.7 (p<1e-4), and replacing the plan outright +13.3 (p=1e-4). Gains are indifferent to the plan's prose style and to its likelihood under the planner, but not to its geometry. Modular planner-decoder designs are therefore viable, provided the plan is internally consistent: box-text contradictions induce object duplication and identity fusion. We release the plan-fidelity evaluation protocol, all plans, and 12k generated images.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 37 of its 38 references that are in the map. First posted on arXiv 22 Aug 2026.
Builds on
37 of 38 references.
References · 37
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
- 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
- 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
- 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
- 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 787
- 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
- 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
- 2023Scaling Open-Vocabulary Object Detectioncited by 335
- 2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 253
- 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 245
- 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
25 more
- 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 162
- 2025Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Stepcited by 130
- 2023Self-correcting LLM-controlled Diffusion Modelscited by 112
- 2025GoT: Unleashing Reasoning Capability of Multimodal Large Language Model for Visual Generation and Editingcited by 103
- 2022Does CLIP Bind Concepts? Probing Compositionality in Large Image Modelscited by 101
- 2022Benchmarking Spatial Relationships in Text-to-Image Generationcited by 96
- 2025GoT-R1: Unleashing Reasoning Capability of MLLM for Visual Generation with Reinforcement Learningcited by 51
- 2025Reflect-DiT: Inference-Time Scaling for Text-to-Image Diffusion Transformers via In-Context Reflectioncited by 42
- 2023Text encoders bottleneck compositionality in contrastive vision-language modelscited by 34
- 2025RealUnify: Do Unified Models Truly Benefit from Unification? A Comprehensive Benchmarkcited by 33
- 2024Getting it Right: Improving Spatial Consistency in Text-to-Image Modelscited by 30
- 2025On the Faithfulness of Visual Thinking: Measurement and Enhancementcited by 17
- 2023Diagnostic Benchmark and Iterative Inpainting for Layout-Guided Image Generationcited by 13
- 2025Evaluating Reasoning Faithfulness in Medical Vision-Language Models using Multimodal Perturbationscited by 9
- 2024Rethinking The Training And Evaluation of Rich-Context Layout-to-Image Generationcited by 8
- 2026UReason: Benchmarking Reasoning-to-Generation Alignment in Unified Multimodal Modelscited by 6
- 2026Think in Strokes, Not Pixels: Process-Driven Image Generation via Interleaved Reasoningcited by 6
- 2023Text-Anchored Score Composition: Tackling Condition Misalignment in Text-to-Image Diffusion Modelscited by 4
- 2025Say One Thing, Do Another? Diagnosing Reasoning-Execution Gaps in VLM-Powered Mobile-Use Agentscited by 4
- 2026Iterative Refinement Improves Compositional Image Generationcited by 4
- 2026SANEval: Open-Vocabulary Compositional Benchmarks with Failure-mode Diagnosiscited by 3
- 2026EPIC: Efficient Predicate-Guided Inference-Time Control for Compositional Text-to-Image Generationcited by 2
- 2026Order Is Not Layout: Order-to-Space Bias in Image Generationcited by 1
- 2026Unlocking Complex Visual Generation via Closed-Loop Verified Reasoningcited by 1
- 2026Text-to-Image Models Need Less from Text Encoders Than You Thinkcited by 1
Built on by
Not cited yet.