# What Makes a Scene ? Scene Graph-based Evaluation and Feedback for Controllable Generation

arXiv: https://arxiv.org/abs/2411.15435 · HTML: https://mlsanity.com/paper/2411.15435/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 69 references that are in the map.
- First posted on arXiv 23 Nov 2024.

## Builds on (most cited first, 30 of 53)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis](https://mlsanity.com/paper/2310.00426/) (2023)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)
- [Compositional Visual Generation with Composable Diffusion Models](https://mlsanity.com/paper/2206.01714/) (2022)

## Built on by (most cited first, 6 of 6)

- [Agent Identity Evals: Measuring Agentic Identity](https://mlsanity.com/paper/2507.17257/) (2025)
- [From Data to Modeling: Fully Open-vocabulary Scene Graph Generation](https://mlsanity.com/paper/2505.20106/) (2025)
- [Integrating Graphs, Large Language Models, and Agents: Reasoning and Retrieval](https://mlsanity.com/paper/2604.15951/) (2026)
- [Synthetic Curriculum Reinforces Compositional Text-to-Image Generation](https://mlsanity.com/paper/2511.18378/) (2025)
- [Self-Correction Can Amplify Hallucinations: Fact-Level Repair with Graph-Based Evidence Routing in Multimodal Generation](https://mlsanity.com/paper/2606.00232/) (2026)
- [Panoptic Scene Program Diffusion Transformer](https://mlsanity.com/paper/2609.31780/) (2026)

## Abstract (arXiv)

While text-to-image generation has been extensively studied, generating images from scene graphs remains relatively underexplored, primarily due to challenges in accurately modeling spatial relationships and object interactions. To fill this gap, we introduce Scene-Bench, a comprehensive benchmark designed to evaluate and enhance the factual consistency in generating natural scenes. Scene-Bench comprises MegaSG, a large-scale dataset of one million images annotated with scene graphs, facilitating the training and fair comparison of models across diverse and complex scenes. Additionally, we propose SGScore, a novel evaluation metric that leverages chain-of-thought reasoning capabilities of multimodal large language models (LLMs) to assess both object presence and relationship accuracy, offering a more effective measure of factual consistency than traditional metrics like FID and CLIPScore. Building upon this evaluation framework, we develop a scene graph feedback pipeline that iteratively refines generated images by identifying and correcting discrepancies between the scene graph and the image. Extensive experiments demonstrate that Scene-Bench provides a more comprehensive and effective evaluation framework compared to existing benchmarks, particularly for complex scene generation. Furthermore, our feedback strategy significantly enhances the factual consistency of image generation models, advancing the field of controllable image generation.