# TP2O: Creative Text Pair-to-Object Generation using Balance Swap-Sampling

arXiv: https://arxiv.org/abs/2310.01819 · HTML: https://mlsanity.com/paper/2310.01819/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 64 references that are in the map.
- First posted on arXiv 3 Oct 2023.

## Builds on (most cited first, 30 of 52)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022)
- [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2305.01569/) (2023)
- [Denoising Diffusion Restoration Models](https://mlsanity.com/paper/2201.11793/) (2022)
- [Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis](https://mlsanity.com/paper/2306.09341/) (2023)
- [Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2301.13826/) (2023)
- [Pseudo Numerical Methods for Diffusion Models on Manifolds](https://mlsanity.com/paper/2202.09778/) (2022)
- [Compositional Visual Generation with Composable Diffusion Models](https://mlsanity.com/paper/2206.01714/) (2022)
- [Variational Autoencoders and Nonlinear ICA: A Unifying Framework](https://mlsanity.com/paper/1907.04809/) (2019)
- [UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Models](https://mlsanity.com/paper/2302.04867/) (2023)
- [Towards Out-Of-Distribution Generalization: A Survey](https://mlsanity.com/paper/2108.13624/) (2021)
- [Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis](https://mlsanity.com/paper/2212.05032/) (2022)
- [DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesis](https://mlsanity.com/paper/1904.01310/) (2019)
- [Learning to Generate Novel Domains for Domain Generalization](https://mlsanity.com/paper/2007.03304/) (2020)
- [Deep Domain-Adversarial Image Generation for Domain Generalisation](https://mlsanity.com/paper/2003.06054/) (2020)
- [SpaText: Spatio-Textual Representation for Controllable Image Generation](https://mlsanity.com/paper/2211.14305/) (2022)

## Built on by (most cited first, 16 of 16)

- [DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models](https://mlsanity.com/paper/2411.15024/) (2024)
- [Novel Object Synthesis via Adaptive Text-Image Harmony](https://mlsanity.com/paper/2410.20823/) (2024)
- [Distribution-Conditional Generation: From Class Distribution to Creative Generation](https://mlsanity.com/paper/2505.03667/) (2025)
- [TCDiff++: An End-to-end Trajectory-Controllable Diffusion Model for Harmonious Music-Driven Group Choreography](https://mlsanity.com/paper/2506.18671/) (2025)
- [VLM-Guided Adaptive Negative Prompting for Creative Generation](https://mlsanity.com/paper/2510.10715/) (2025)
- [Redefining <Creative> in Dictionary: Towards an Enhanced Semantic Understanding of Creative Generation](https://mlsanity.com/paper/2410.24160/) (2024)
- [Category-Aware 3D Object Composition with Disentangled Texture and Shape Multi-view Diffusion](https://mlsanity.com/paper/2509.02357/) (2025)
- [AGSwap: Overcoming Category Boundaries in Object Fusion via Adaptive Group Swapping](https://mlsanity.com/paper/2509.18699/) (2025)
- [Chirpy3D: Part-Aware Multi-View Diffusion for Creative Fine-Grained Object Generation](https://mlsanity.com/paper/2501.04144/) (2025)
- [GuessBench: Sensemaking Multimodal Creativity in the Wild](https://mlsanity.com/paper/2506.00814/) (2025)
- [VMDiff: Visual Mixing Diffusion for Limitless Cross-Object Synthesis](https://mlsanity.com/paper/2509.23605/) (2025)
- [RMLer: Synthesizing Novel Objects across Diverse Categories via Reinforcement Mixing Learning](https://mlsanity.com/paper/2512.19300/) (2025)
- [Blending Concepts with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2506.23630/) (2025)
- [Imagine for Me: Creative Conceptual Blending of Real Images and Text via Blended Attention](https://mlsanity.com/paper/2506.24085/) (2025)
- [A Creative Agent is Worth a 64-Token Template](https://mlsanity.com/paper/2603.17895/) (2026)
- [Self-Creative Text-to-Object Generation using Semantic-Aware Spatial Weighting](https://mlsanity.com/paper/2605.19554/) (2026)

## Abstract (arXiv)

Generating creative combinatorial objects from two seemingly unrelated object texts is a challenging task in text-to-image synthesis, often hindered by a focus on emulating existing data distributions. In this paper, we develop a straightforward yet highly effective method, called balance swap-sampling. First, we propose a swapping mechanism that generates a novel combinatorial object image set by randomly exchanging intrinsic elements of two text embeddings through a cutting-edge diffusion model. Second, we introduce a balance swapping region to efficiently sample a small subset from the newly generated image set by balancing CLIP distances between the new images and their original generations, increasing the likelihood of accepting the high-quality combinations. Last, we employ a segmentation method to compare CLIP distances among the segmented components, ultimately selecting the most promising object from the sampled subset. Extensive experiments demonstrate that our approach outperforms recent SOTA T2I methods. Surprisingly, our results even rival those of human artists, such as frog-broccoli.