Unified Personalized Understanding, Generating and Editing
On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning
Abstract (arXiv)
Unified large multimodal models (LMMs) have achieved remarkable progress in general-purpose multimodal understanding and generation. However, they still operate under a ``one-size-fits-all'' paradigm and struggle to model user-specific concepts (e.g., generate a photo of texttt<maeve>) in a consistent and controllable manner. Existing personalization methods typically rely on external retrieval, which is inefficient and poorly integrated into unified multimodal pipelines. Recent personalized unified models introduce learnable soft prompts to encode concept information, yet they either couple understanding and generation or depend on complex multi-stage training, leading to cross-task interference and ultimately to fuzzy or misaligned personalized knowledge. We present OmniPersona, an end-to-end personalization framework for unified LMMs that, for the first time, integrates personalized understanding, generation, and image editing within a single architecture. OmniPersona introduces structurally decoupled concept tokens, allocating dedicated subspaces for different tasks to minimize interference, and incorporates an explicit knowledge replay mechanism that propagates personalized attribute knowledge across tasks, enabling consistent personalized behavior. To systematically evaluate unified personalization, we propose textbftextttOmniPBench, extending the public UnifyBench concept set with personalized editing tasks and cross-task evaluation protocols integrating understanding, generation, and editing. Experimental results demonstrate that OmniPersona delivers competitive and robust performance across diverse personalization tasks. We hope OmniPersona will serve as a strong baseline and spur further research on controllable, unified personalization.
Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 40 references that are in the map. First posted on arXiv 11 Jan 2026.
Builds on
39 of 40 references.
Closest ancestors · 5
Same direction, newest first
- 2025PixelRefer: A Unified Framework for Spatio-Temporal Object Referring with Arbitrary Granularity27 Oct
- 2025YoChameleon: Personalized Vision and Language Generation29 Apr
- 2024RAP: Retrieval-Augmented Personalization for Multimodal Large Language Models17 Oct
- 2024Yo'LLaVA: Your Personalized Language and Vision Assistant13 Jun
- 2024HyperLLaVA: Dynamic Visual and Language Expert Tuning for Multimodal Large Language Models20 Mar
Other references · 34
- 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
- 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
- 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,464
- 2025Qwen-Image Technical Reportcited by 1,044
- 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
- 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
- 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
- 2024Emu3: Next-Token Prediction is All You Needcited by 781
- 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
- 2023LaMP: When Large Language Models Meet Personalizationcited by 437
- 2025BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Datasetcited by 376
- 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 345
22 more
- 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
- 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
- 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 295
- 2025Show-o2: Improved Native Unified Multimodal Modelscited by 250
- 2025Transfer between Modalities with MetaQueriescited by 236
- 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 206
- 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 153
- 2025HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptationcited by 114
- 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 85
- 2024VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLMcited by 72
- 2024MyVLM: Personalizing VLMs for User-Specific Queriescited by 58
- 2025UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokenscited by 56
- 2022BoostMIS: Boosting Medical Image Semi-supervised Learning with Adaptive Pseudo Labeling and Informative Active Annotationcited by 54
- 2024MC-LLaVA: Multi-Concept Personalized Vision-Language Modelcited by 54
- 2025UniToken: Harmonizing Multimodal Understanding and Generation through Unified Visual Encodingcited by 49
- 2022DUET: A Tuning-Free Device-Cloud Collaborative Parameters Generation Framework for Efficient Device Model Generalizationcited by 35
- 2024Imagine yourself: Tuning-Free Personalized Image Generationcited by 30
- 2021Consensus Graph Representation Learning for Better Grounded Image Captioningcited by 28
- 2025EOC-Bench: Can MLLMs Identify, Recall, and Forecast Objects in an Egocentric World?cited by 21
- 2023Revisiting the Domain Shift and Sample Uncertainty in Multi-source Active Domain Transfercited by 20
- 2025WiseEdit: Benchmarking Cognition- and Creativity-Informed Image Editingcited by 11
- 2024MAKIMA: Tuning-free Multi-Attribute Open-domain Video Editing via Mask-Guided Attention Modulationcited by 2
Built on by
Citing papers per year: 2026: 8.
8 papers.
- 2026PEARL: Personalized Streaming Video Understanding Modelcited by 9
- 2026CoCo: Code as CoT for Text-to-Image Preview and Rare Concept Generationcited by 6
- 2026InstructSAM: Segment Any Instance with Any Instructionscited by 6
- 2026VisualThink-VLA: Visual Intermediate Reasoning for Effective and Low-Latency Vision-Language-Action Policiescited by 4
- 2026Regulating Anatomy-Aware Rewards via Trajectory-Integral Feedback for Volumetric Computed Tomography Analysiscited by 2
- 2026Best of Both Worlds: Multimodal Reasoning and Generation via Unified Discrete Flow Matchingcited by 1
- 2026EgoCoT-Bench: Benchmarking Grounded and Verifiable Operation-Centric Chain of Thought Reasoning for MLLMscited by 0
- 2026Where a New Concept Must Enter: Entry Point Gates Cross-Task Usability in Unified Multimodal Modelscited by 0
- 2026Where a New Concept Must Enter: Entry Point Gates Cross-Task Usability in Unified Multimodal Models18 Aug 2026
- 2026VisualThink-VLA: Visual Intermediate Reasoning for Effective and Low-Latency Vision-Language-Action Policies28 May 2026
- 2026InstructSAM: Segment Any Instance with Any Instructions25 May 2026
- 2026Regulating Anatomy-Aware Rewards via Trajectory-Integral Feedback for Volumetric Computed Tomography Analysis19 May 2026
- 2026EgoCoT-Bench: Benchmarking Grounded and Verifiable Operation-Centric Chain of Thought Reasoning for MLLMs19 May 2026
- 2026PEARL: Personalized Streaming Video Understanding Model20 Mar 2026
- 2026CoCo: Code as CoT for Text-to-Image Preview and Rare Concept Generation9 Mar 2026
- 2026Best of Both Worlds: Multimodal Reasoning and Generation via Unified Discrete Flow Matching12 Feb 2026
Similar, not cited
Close in wording, no citation either way
- 2025MM-R1: Unleashing the Power of Unified Multimodal Large Language Models for Personalized Image Generation
- 2025OmniMamba: Efficient and Unified Multimodal Understanding and Generation via State Space Models
- 2026Omni-R1: Towards the Unified Generative Paradigm for Multimodal Reasoning
- 2025MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer
- 2025UniModel: A Visual-Only Framework for Unified Multimodal Understanding and Generation
- 2026Context Unrolling in Omni Models
- 2026UniRect-CoT: Enhancing Generation in Unified Multimodal Models via Reflective Rectification with Inherent Understanding
- 2026Omni-Persona: Systematic Benchmarking and Improving Omnimodal Personalization