# Personalized Image Generation with Large Multimodal Models

arXiv: https://arxiv.org/abs/2410.14170 · HTML: https://mlsanity.com/paper/2410.14170/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 10 of its 57 references that are in the map.
- First posted on arXiv 18 Oct 2024.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 10 of 10)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models](https://mlsanity.com/paper/2402.17177/) (2024)
- [Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization](https://mlsanity.com/paper/2402.03161/) (2024)
- [LongLaMP: A Benchmark for Personalized Long-form Text Generation](https://mlsanity.com/paper/2407.11016/) (2024)
- [Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Efforts](https://mlsanity.com/paper/2406.10471/) (2024)
- [Generative Recommendation: Towards Next-generation Recommender Paradigm](https://mlsanity.com/paper/2304.03516/) (2023)
- [Imagine yourself: Tuning-Free Personalized Image Generation](https://mlsanity.com/paper/2409.13346/) (2024)
- [AdBooster: Personalized Ad Creative Generation using Stable Diffusion Outpainting](https://mlsanity.com/paper/2309.11507/) (2023)

## Built on by (most cited first, 15 of 15)

- [SPEED: Scalable, Precise, and Efficient Concept Erasure for Diffusion Models](https://mlsanity.com/paper/2503.07392/) (2025)
- [Easier Painting Than Thinking: Can Text-to-Image Models Set the Stage, but Not Direct the Play?](https://mlsanity.com/paper/2509.03516/) (2025)
- [Personalized Image Editing in Text-to-Image Diffusion Models via Collaborative Direct Preference Optimization](https://mlsanity.com/paper/2511.05616/) (2025)
- [Draw Your Mind: Personalized Generation via Condition-Level Modeling in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2508.03481/) (2025)
- [Personalized Reward Modeling for Text-to-Image Generation](https://mlsanity.com/paper/2511.19458/) (2025)
- [Thinking with Frames: Generative Video Distortion Evaluation via Frame Reward Model](https://mlsanity.com/paper/2601.04033/) (2026)
- [Premier: Personalized Preference Modulation with Learnable User Embedding in Text-to-Image Generation](https://mlsanity.com/paper/2603.20725/) (2026)
- [Preference-Aware Rubric Learning for Personalized Evaluation](https://mlsanity.com/paper/2605.31545/) (2026)
- [Design Your Ad: Personalized Advertising Image and Text Generation with Unified Autoregressive Models](https://mlsanity.com/paper/2605.12138/) (2026)
- [PIPBench: A Profile-Inclusive Framework for Personalized Image Generation Evaluation](https://mlsanity.com/paper/2607.06440/) (2026)
- [Dynamic Training-Free Fusion of Subject and Style LoRAs](https://mlsanity.com/paper/2602.15539/) (2026)
- [Gate-and-Merge: Zero-shot Compositional Personalization of Vision Language Models](https://mlsanity.com/paper/2605.08702/) (2026)
- [ZIPP:Zero-shot Image Personalization from Personas](https://mlsanity.com/paper/2606.08841/) (2026)
- [Personalized Image Generation with Reasoning and Reflection](https://mlsanity.com/paper/2610.00737/) (2026)
- [Does Every User Need a Private LoRA? Decoupling Personalization from Per-User Adaptation](https://mlsanity.com/paper/2610.02353/) (2026)

## Abstract (arXiv)

Personalized content filtering, such as recommender systems, has become a critical infrastructure to alleviate information overload. However, these systems merely filter existing content and are constrained by its limited diversity, making it difficult to meet users' varied content needs. To address this limitation, personalized content generation has emerged as a promising direction with broad applications. Nevertheless, most existing research focuses on personalized text generation, with relatively little attention given to personalized image generation. The limited work in personalized image generation faces challenges in accurately capturing users' visual preferences and needs from noisy user-interacted images and complex multimodal instructions. Worse still, there is a lack of supervised data for training personalized image generation models. To overcome the challenges, we propose a Personalized Image Generation Framework named Pigeon, which adopts exceptional large multimodal models with three dedicated modules to capture users' visual preferences and needs from noisy user history and multimodal instructions. To alleviate the data scarcity, we introduce a two-stage preference alignment scheme, comprising masked preference reconstruction and pairwise preference alignment, to align Pigeon with the personalized image generation task. We apply Pigeon to personalized sticker and movie poster generation, where extensive quantitative results and human evaluation highlight its superiority over various generative baselines.