# Improving Physical Object State Representation in Text-to-Image Generative Systems

arXiv: https://arxiv.org/abs/2505.02236 · HTML: https://mlsanity.com/paper/2505.02236/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 57 references that are in the map.
- First posted on arXiv 4 May 2025.

## Builds on (most cited first, 26 of 26)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling](https://mlsanity.com/paper/2501.17811/) (2025)
- [Diffusion Model Alignment Using Direct Preference Optimization](https://mlsanity.com/paper/2311.12908/) (2023)
- [Emu3: Next-Token Prediction is All You Need](https://mlsanity.com/paper/2409.18869/) (2024)
- [Evaluating Text-to-Visual Generation with Image-to-Text Generation](https://mlsanity.com/paper/2404.01291/) (2024)
- [OmniGen: Unified Image Generation](https://mlsanity.com/paper/2409.11340/) (2024)
- [Diffusion Policy Policy Optimization](https://mlsanity.com/paper/2409.00588/) (2024)
- [GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generation](https://mlsanity.com/paper/2406.13743/) (2024)
- [Self-correcting LLM-controlled Diffusion Models](https://mlsanity.com/paper/2311.16090/) (2023)
- [Vision-Language Models Do Not Understand Negation](https://mlsanity.com/paper/2501.09425/) (2025)
- [Commonsense-T2I Challenge: Can Text-to-Image Generation Models Understand Commonsense?](https://mlsanity.com/paper/2406.07546/) (2024)
- [Learn "No" to Say "Yes" Better: Improving Vision-Language Models via Negations](https://mlsanity.com/paper/2403.20312/) (2024)
- [Learning Object State Changes in Videos: An Open-World Perspective](https://mlsanity.com/paper/2312.11782/) (2023)
- [Correcting Diffusion Generation through Resampling](https://mlsanity.com/paper/2312.06038/) (2023)
- [Stable Diffusion Dataset Generation for Downstream Classification Tasks](https://mlsanity.com/paper/2405.02698/) (2024)
- [Do Pre-trained Vision-Language Models Encode Object States?](https://mlsanity.com/paper/2409.10488/) (2024)
- [Improving Explicit Spatial Relationships in Text-to-Image Generation through an Automatically Derived Dataset](https://mlsanity.com/paper/2403.00587/) (2024)

## Abstract (arXiv)

Current text-to-image generative models struggle to accurately represent object states (e.g., "a table without a bottle," "an empty tumbler"). In this work, we first design a fully-automatic pipeline to generate high-quality synthetic data that accurately captures objects in varied states. Next, we fine-tune several open-source text-to-image models on this synthetic data. We evaluate the performance of the fine-tuned models by quantifying the alignment of the generated images to their prompts using GPT4o-mini, and achieve an average absolute improvement of 8+% across four models on the public GenAI-Bench dataset. We also curate a collection of 200 prompts with a specific focus on common objects in various physical states. We demonstrate a significant improvement of an average of 24+% over the baseline on this dataset. We release all evaluation prompts and code.