# Can Pre-Trained Text-to-Image Models Generate Visual Goals for Reinforcement Learning?

arXiv: https://arxiv.org/abs/2307.07837 · HTML: https://mlsanity.com/paper/2307.07837/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jul 2023.

## Built on by (most cited first, 20 of 20)

- [Diffusion Models for Reinforcement Learning: A Survey](https://mlsanity.com/paper/2311.01223/) (2023)
- [General Flow as Foundation Affordance for Scalable Robot Learning](https://mlsanity.com/paper/2401.11439/) (2024)
- [Robo-ABC: Affordance Generalization Beyond Categories via Semantic Correspondence for Robot Manipulation](https://mlsanity.com/paper/2401.07487/) (2024)
- [Boosting Continuous Control with Consistency Policy](https://mlsanity.com/paper/2310.06343/) (2023)
- [View-Invariant Policy Learning via Zero-Shot Novel View Synthesis](https://mlsanity.com/paper/2409.03685/) (2024)
- [Getting it Right: Improving Spatial Consistency in Text-to-Image Models](https://mlsanity.com/paper/2404.01197/) (2024)
- [Empowering Embodied Visual Tracking with Visual Foundation Models and Offline RL](https://mlsanity.com/paper/2404.09857/) (2024)
- [SAM-E: Leveraging Visual Foundation Model with Sequence Imitation for Embodied Manipulation](https://mlsanity.com/paper/2405.19586/) (2024)
- [GHIL-Glue: Hierarchical Control with Filtered Subgoal Images](https://mlsanity.com/paper/2410.20018/) (2024)
- [Do Visual Imaginations Improve Vision-and-Language Navigation Agents?](https://mlsanity.com/paper/2503.16394/) (2025)
- [TeViR: Text-to-Video Reward with Diffusion Models for Efficient Reinforcement Learning](https://mlsanity.com/paper/2505.19769/) (2025)
- [An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Tasks](https://mlsanity.com/paper/2502.04773/) (2025)
- [Stem-OB: Generalizable Visual Imitation Learning with Stem-Like Convergent Observation through Diffusion Inversion](https://mlsanity.com/paper/2411.04919/) (2024)
- [WebRenderBench: Enhancing Web Interface Generation through Layout-Style Consistency and Reinforcement Learning](https://mlsanity.com/paper/2510.04097/) (2025)
- [ActionSink: Toward Precise Robot Manipulation with Dynamic Integration of Action Flow](https://mlsanity.com/paper/2508.03218/) (2025)
- [MVR: Multi-view Video Reward Shaping for Reinforcement Learning](https://mlsanity.com/paper/2603.01694/) (2026)
- [From Goal-Conditioned to Language-Conditioned Agents via Vision-Language Models](https://mlsanity.com/paper/2409.16024/) (2024)
- [PACA: Perspective-Aware Cross-Attention Representation for Zero-Shot Scene Rearrangement](https://mlsanity.com/paper/2410.22059/) (2024)
- [Rhetorical Text-to-Image Generation via Two-layer Diffusion Policy Optimization](https://mlsanity.com/paper/2505.22792/) (2025)
- [EVLP:Learning Unified Embodied Vision-Language Planner with Reinforced Supervised Fine-Tuning](https://mlsanity.com/paper/2511.05553/) (2025)

## Abstract (arXiv)

Pre-trained text-to-image generative models can produce diverse, semantically rich, and realistic images from natural language descriptions. Compared with language, images usually convey information with more details and less ambiguity. In this study, we propose Learning from the Void (LfVoid), a method that leverages the power of pre-trained text-to-image models and advanced image editing techniques to guide robot learning. Given natural language instructions, LfVoid can edit the original observations to obtain goal images, such as "wiping" a stain off a table. Subsequently, LfVoid trains an ensembled goal discriminator on the generated image to provide reward signals for a reinforcement learning agent, guiding it to achieve the goal. The ability of LfVoid to learn with zero in-domain training on expert demonstrations or true goal observations (the void) is attributed to the utilization of knowledge from web-scale generative models. We evaluate LfVoid across three simulated tasks and validate its feasibility in the corresponding real-world scenarios. In addition, we offer insights into the key considerations for the effective integration of visual generative models into robot learning workflows. We posit that our work represents an initial step towards the broader application of pre-trained visual generative models in the robotics field. Our project page: https://lfvoid-rl.github.io/.