# Can Text-to-Image Models Draw from the Right Frame of Reference?

arXiv: https://arxiv.org/abs/2608.03357 · HTML: https://mlsanity.com/paper/2608.03357/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 34 references that are in the map.
- First posted on arXiv 4 Aug 2026.

## Builds on (most cited first, 16 of 16)

- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](https://mlsanity.com/paper/2310.11513/) (2023)
- [TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering](https://mlsanity.com/paper/2303.11897/) (2023)
- [Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis](https://mlsanity.com/paper/2412.04431/) (2024)
- [Text-to-image Diffusion Models in Generative AI: A Survey](https://mlsanity.com/paper/2303.07909/) (2023)
- [Seedream 4.0: Toward Next-generation Multimodal Image Generation](https://mlsanity.com/paper/2509.20427/) (2025)
- [Understanding World or Predicting Future? A Comprehensive Survey of World Models](https://mlsanity.com/paper/2411.14499/) (2024)
- [OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation](https://mlsanity.com/paper/2506.07977/) (2025)
- [Controllable Generation with Text-to-Image Diffusion Models: A Survey](https://mlsanity.com/paper/2403.04279/) (2024)
- [TIIF-Bench: How Does Your T2I Model Follow Your Instructions?](https://mlsanity.com/paper/2506.02161/) (2025)
- [Qwen-Image-2.0 Technical Report](https://mlsanity.com/paper/2605.10730/) (2026)
- [T2I-Copilot: A Training-Free Multi-Agent Text-to-Image System for Enhanced Prompt Interpretation and Interactive Generation](https://mlsanity.com/paper/2507.20536/) (2025)
- [FoREST: Frame of Reference Evaluation in Spatial Reasoning Tasks](https://mlsanity.com/paper/2502.17775/) (2025)
- [GenSpace: Benchmarking Spatially-Aware Image Generation](https://mlsanity.com/paper/2505.24870/) (2025)
- [Allocentric Perceiver: Disentangling Allocentric Reasoning from Egocentric Visual Priors via Frame Instantiation](https://mlsanity.com/paper/2602.05789/) (2026)
- [VisionDirector: Vision-Language Guided Closed-Loop Refinement for Generative Image Synthesis](https://mlsanity.com/paper/2512.19243/) (2025)

## Abstract (arXiv)

Spatial instruction following has become a crucial requirement for text-to-image (T2I) generation. A common challenge arises when directional expressions are interpreted under different frames of reference. For example, ``the left of'' may refer to the viewer's image coordinates or to the intrinsic orientation of an object, leading to different expected layouts. Existing T2I benchmarks reveal important layout failures, yet they rarely isolate whether models can follow a specified frame of reference when it differs from camera view. To mitigate this gap, we introduce FoR-T2I, a benchmark for evaluating this distinction with 1,200 prompt pairs built from controlled spatial layouts. In each pair, the camera-view (Cam) prompt states the target relation in camera view, while the frame-of-reference (FoR) prompt describes the same target placement through an oriented anchor object. Across 22 closed-source and open-source T2I models, mean final accuracy is 41.8% lower on FoR prompts than on matched Cam prompts; even the best-performing model achieves only 44.3% FoR accuracy. This suggests that current models struggle more when the same layout is described through an object's orientation rather than directly in image coordinates. We further analyze this gap by relation type and camera view, compare several training-free prompting and feedback-based mitigation strategies, and propose a VLM-gated rewriting approach that selects rewritten prompts using visual feedback, improving average FoR accuracy from 25.0% to 29.2% under the same generation budget.