# Human-Guided Image Generation for Expanding Small-Scale Training Image Datasets

arXiv: https://arxiv.org/abs/2412.16839 · HTML: https://mlsanity.com/paper/2412.16839/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 70 references that are in the map.
- First posted on arXiv 22 Dec 2024.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Rethinking FID: Towards a Better Evaluation Metric for Image Generation](https://mlsanity.com/paper/2401.09603/) (2023)
- [Effective Data Augmentation With Diffusion Models](https://mlsanity.com/paper/2302.07944/) (2023)
- [Diversify Your Vision Datasets with Automatic Diffusion-Based Augmentation](https://mlsanity.com/paper/2305.16289/) (2023)
- [Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Models](https://mlsanity.com/paper/2304.09337/) (2023)
- [Expanding Small-Scale Datasets with Guided Imagination](https://mlsanity.com/paper/2211.13976/) (2022)
- [PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement](https://mlsanity.com/paper/2403.04014/) (2024)
- [PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation](https://mlsanity.com/paper/2307.09036/) (2023)
- [Towards Better Analysis of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1604.07043/) (2016)
- [RePrompt: Automatic Prompt Editing to Refine AI-Generative Art Towards Precise Expressions](https://mlsanity.com/paper/2302.09466/) (2023)
- [PromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactions](https://mlsanity.com/paper/2308.05184/) (2023)
- [OoDAnalyzer: Interactive Analysis of Out-of-Distribution Samples](https://mlsanity.com/paper/2002.03103/) (2020)
- [Foundation Models Meet Visualizations: Challenges and Opportunities](https://mlsanity.com/paper/2310.05771/) (2023)
- [Distribution-Aware Data Expansion with Diffusion Models](https://mlsanity.com/paper/2403.06741/) (2024)
- [Semantic-Guided Generative Image Augmentation Method with Diffusion Models for Image Classification](https://mlsanity.com/paper/2302.02070/) (2023)
- [Memory-based Jitter: Improving Visual Recognition on Long-tailed Data with Diversity In Memory](https://mlsanity.com/paper/2008.09809/) (2020)
- [Diagnosing Ensemble Few-Shot Classifiers](https://mlsanity.com/paper/2206.04372/) (2022)
- [PrompTHis: Visualizing the Process and Influence of Prompt Editing during Text-to-Image Creation](https://mlsanity.com/paper/2403.09615/) (2024)
- [Towards Visual Explainable Active Learning for Zero-Shot Classification](https://mlsanity.com/paper/2108.06730/) (2021)
- [On the Limitation of Diffusion Models for Synthesizing Training Datasets](https://mlsanity.com/paper/2311.13090/) (2023)
- [Advancing Text-Driven Chest X-Ray Generation with Policy-Based Reinforcement Learning](https://mlsanity.com/paper/2403.06516/) (2024)
- [A Unified Interactive Model Evaluation for Classification, Object Detection, and Instance Segmentation in Computer Vision](https://mlsanity.com/paper/2308.05168/) (2023)
- [VideoPro: A Visual Analytics Approach for Interactive Video Programming](https://mlsanity.com/paper/2308.00401/) (2023)
- [ModalChorus: Visual Probing and Alignment of Multi-modal Embeddings via Modal Fusion Map](https://mlsanity.com/paper/2407.12315/) (2024)
- [Cluster-Aware Grid Layout](https://mlsanity.com/paper/2308.03651/) (2023)

## Built on by (most cited first, 10 of 10)

- [IMAGGarment: Fine-Grained Garment Generation for Controllable Fashion Design](https://mlsanity.com/paper/2504.13176/) (2025)
- [Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlook](https://mlsanity.com/paper/2503.18016/) (2025)
- [DriveGEN: Generalized and Robust 3D Detection in Driving via Controllable Text-to-Image Diffusion Generation](https://mlsanity.com/paper/2503.11122/) (2025)
- [Beyond Isolated Words: Diffusion Brush for Handwritten Text-Line Generation](https://mlsanity.com/paper/2508.03256/) (2025)
- [MoSa: Motion Generation with Scalable Autoregressive Modeling](https://mlsanity.com/paper/2511.01200/) (2025)
- [Interactive Hybrid Rice Breeding with Parametric Dual Projection](https://mlsanity.com/paper/2507.11848/) (2025)
- [Enhancing Small-Scale Dataset Expansion with Triplet-Connection-based Sample Re-Weighting](https://mlsanity.com/paper/2508.07723/) (2025)
- [Look, Compare and Draw: Differential Query Transformer for Automatic Oil Painting](https://mlsanity.com/paper/2603.27720/) (2026)
- [Compositional World Knowledge leads to High Utility Synthetic data](https://mlsanity.com/paper/2503.04687/) (2025)
- [Human-Guided Causal Knowledge Injection for Virtual Cells](https://mlsanity.com/paper/2608.08430/) (2026)

## Abstract (arXiv)

The performance of computer vision models in certain real-world applications (e.g., rare wildlife observation) is limited by the small number of available images. Expanding datasets using pre-trained generative models is an effective way to address this limitation. However, since the automatic generation process is uncontrollable, the generated images are usually limited in diversity, and some of them are undesired. In this paper, we propose a human-guided image generation method for more controllable dataset expansion. We develop a multi-modal projection method with theoretical guarantees to facilitate the exploration of both the original and generated images. Based on the exploration, users refine the prompts and re-generate images for better performance. Since directly refining the prompts is challenging for novice users, we develop a sample-level prompt refinement method to make it easier. With this method, users only need to provide sample-level feedback (e.g., which samples are undesired) to obtain better prompts. The effectiveness of our method is demonstrated through the quantitative evaluation of the multi-modal projection method, improved model performance in the case study for both classification and object detection tasks, and positive feedback from the experts.