# Selecting Fine-Tuning Examples by Quizzing VLMs

arXiv: https://arxiv.org/abs/2511.12002 · HTML: https://mlsanity.com/paper/2511.12002/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 19 of its 23 references that are in the map.
- First posted on arXiv 15 Nov 2025.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 19 of 19)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering](https://mlsanity.com/paper/2303.11897/) (2023)
- [Multimodal datasets: misogyny, pornography, and malignant stereotypes](https://mlsanity.com/paper/2110.01963/) (2021)
- [Human Preference Score: Better Aligning Text-to-Image Models with Human Preference](https://mlsanity.com/paper/2303.14420/) (2023)
- [Stable Bias: Analyzing Societal Representations in Diffusion Models](https://mlsanity.com/paper/2303.11408/) (2023)
- [The Bias Amplification Paradox in Text-to-Image Generation](https://mlsanity.com/paper/2308.00755/) (2023)
- [A Survey on Quality Metrics for Text-to-Image Generation](https://mlsanity.com/paper/2403.11821/) (2024)
- [DiffuseKronA: A Parameter Efficient Fine-tuning Method for Personalized Diffusion Models](https://mlsanity.com/paper/2402.17412/) (2024)
- [CuRe: Cultural Gaps in the Long Tail of Text-to-Image Systems](https://mlsanity.com/paper/2506.08071/) (2025)
- [Evaluating Text-to-Image Generative Models: An Empirical Study on Human Image Synthesis](https://mlsanity.com/paper/2403.05125/) (2024)
- [When Image Generation Goes Wrong: A Safety Analysis of Stable Diffusion Models](https://mlsanity.com/paper/2411.15516/) (2024)
- [Transformed Low-rank Adaptation via Tensor Decomposition and Its Applications to Text-to-image Models](https://mlsanity.com/paper/2501.08727/) (2025)
- [Imagine a dragon made of seaweed: How images enhance learning in Wikipedia](https://mlsanity.com/paper/2403.07613/) (2024)

## Abstract (arXiv)

A challenge in fine-tuning text-to-image diffusion models for specific topics is to select good examples. Fine-tuning from image sets of varying quality, such as Wikipedia Commons, will often produce poor output. However, training images that textitdo exemplify the target concept (e.g., a textitfemale Mountain Bluebird) help ensure that the generated images are similarly representative (e.g., have the prototypical blue-wings and gray chest). In this work, we propose QZLoRA, a framework to select images for low-rank adaptation (LoRA). The approach leverages QuizRank, a method to automatically rank images by treating them as an `educational intervention' and `quizzing' a VLM. We demonstrate that QZLoRA can produce better aligned, photorealistic images with fewer samples. We also show that these fine-tuned models can produce stylized that are similarly representative (i.e., illustrations). Our results highlight the promise of combining automated visual reasoning with parameter-efficient fine-tuning for topic-adaptive generative modeling.