# UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokens

arXiv: https://arxiv.org/abs/2505.14671 · HTML: https://mlsanity.com/paper/2505.14671/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- Builds on 68 of its 84 references that are in the map.
- First posted on arXiv 20 May 2025.

## Builds on (most cited first, 30 of 68)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [Diffusion Models in Vision: A Survey](https://mlsanity.com/paper/2209.04747/) (2022)
- [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](https://mlsanity.com/paper/2310.11513/) (2023)
- [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://mlsanity.com/paper/2405.09818/) (2024)
- [Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling](https://mlsanity.com/paper/2501.17811/) (2025)
- [Emu3: Next-Token Prediction is All You Need](https://mlsanity.com/paper/2409.18869/) (2024)
- [Show-o: One Single Transformer to Unify Multimodal Understanding and Generation](https://mlsanity.com/paper/2408.12528/) (2024)
- [Phi-4 Technical Report](https://mlsanity.com/paper/2412.08905/) (2024)
- [LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models](https://mlsanity.com/paper/2407.07895/) (2024)
- [MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?](https://mlsanity.com/paper/2403.14624/) (2024)
- [Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model](https://mlsanity.com/paper/2408.11039/) (2024)
- [Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation](https://mlsanity.com/paper/2410.13848/) (2024)
- [InstantID: Zero-shot Identity-Preserving Generation in Seconds](https://mlsanity.com/paper/2401.07519/) (2024)
- [SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation](https://mlsanity.com/paper/2404.14396/) (2024)
- [VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning](https://mlsanity.com/paper/2504.08837/) (2025)
- [DreamLLM: Synergistic Multimodal Comprehension and Creation](https://mlsanity.com/paper/2309.11499/) (2023)
- [MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?](https://mlsanity.com/paper/2408.13257/) (2024)

## Built on by (most cited first, 30 of 56)

- [WoW: Towards a World omniscient World model Through Embodied Interaction](https://mlsanity.com/paper/2509.22642/) (2025)
- [Reconstruction Alignment Improves Unified Multimodal Models](https://mlsanity.com/paper/2509.07295/) (2025)
- [Latent Sketchpad: Sketching Visual Thoughts to Elicit Multimodal Reasoning in MLLMs](https://mlsanity.com/paper/2510.24514/) (2025)
- [UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learning](https://mlsanity.com/paper/2510.13515/) (2025)
- [Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning](https://mlsanity.com/paper/2510.23473/) (2025)
- [VABench: A Comprehensive Benchmark for Audio-Video Generation](https://mlsanity.com/paper/2512.09299/) (2025)
- [Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain](https://mlsanity.com/paper/2510.17801/) (2025)
- [M2A: Multimodal Memory Agent with Dual-Layer Hybrid Memory for Long-Term Personalized Interactions](https://mlsanity.com/paper/2602.07624/) (2026)
- [Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes](https://mlsanity.com/paper/2510.19400/) (2025)
- [SCALAR: Scale-wise Controllable Visual Autoregressive Learning](https://mlsanity.com/paper/2507.19946/) (2025)
- [Rethinking Driving World Model as Synthetic Data Generator for Perception Tasks](https://mlsanity.com/paper/2510.19195/) (2025)
- [Dr.V: A Hierarchical Perception-Temporal-Cognition Framework to Diagnose Video Hallucination by Fine-grained Spatial-Temporal Grounding](https://mlsanity.com/paper/2509.11866/) (2025)
- [GENIUS: Generative Fluid Intelligence Evaluation Suite](https://mlsanity.com/paper/2602.11144/) (2026)
- [IAG: Input-aware Backdoor Attack on VLM-based Visual Grounding](https://mlsanity.com/paper/2508.09456/) (2025)
- [PeRL: Permutation-Enhanced Reinforcement Learning for Interleaved Vision-Language Reasoning](https://mlsanity.com/paper/2506.14907/) (2025)
- [Text2SQL-Flow: A Robust SQL-Aware Data Augmentation Framework for Text-to-SQL](https://mlsanity.com/paper/2511.10192/) (2025)
- [Does Understanding Inform Generation in Unified Multimodal Models? From Analysis to Path Forward](https://mlsanity.com/paper/2511.20561/) (2025)
- [MICo-150K: A Comprehensive Dataset Advancing Multi-Image Composition](https://mlsanity.com/paper/2512.07348/) (2025)
- [Scone: Bridging Composition and Distinction in Subject-Driven Image Generation via Unified Understanding-Generation Modeling](https://mlsanity.com/paper/2512.12675/) (2025)
- [From Illusion to Intention: Visual Rationale Learning for Vision-Language Reasoning](https://mlsanity.com/paper/2511.23031/) (2025)
- [PEARL: Personalized Streaming Video Understanding Model](https://mlsanity.com/paper/2603.20422/) (2026)
- [UniEdit-I: Training-free Image Editing for Unified VLM via Iterative Understanding, Editing and Verifying](https://mlsanity.com/paper/2508.03142/) (2025)
- [Unified Personalized Understanding, Generating and Editing](https://mlsanity.com/paper/2601.06965/) (2026)
- [Unveiling the Cognitive Compass: Theory-of-Mind-Guided Multimodal Emotion Reasoning](https://mlsanity.com/paper/2602.00971/) (2026)
- [Faithful-First Reasoning, Planning, and Acting for Multimodal LLMs](https://mlsanity.com/paper/2511.08409/) (2025)
- [Semantic Context Matters: Improving Conditioning for Autoregressive Models](https://mlsanity.com/paper/2511.14063/) (2025)
- [Eevee: Towards Close-up High-resolution Video-based Virtual Try-on](https://mlsanity.com/paper/2511.18957/) (2025)
- [CoCo: Code as CoT for Text-to-Image Preview and Rare Concept Generation](https://mlsanity.com/paper/2603.08652/) (2026)
- [GUI-GenBench: Evaluating Image Generation Models as Interactive GUI Environments](https://mlsanity.com/paper/2602.09007/) (2026)
- [Understanding vs. Generation: Navigating Optimization Dilemma in Multimodal Models](https://mlsanity.com/paper/2602.15772/) (2026)

## Abstract (arXiv)

Personalized models have demonstrated remarkable success in understanding and generating concepts provided by users. However, existing methods use separate concept tokens for understanding and generation, treating these tasks in isolation. This may result in limitations for generating images with complex prompts. For example, given the concept langle borangle, generating "langle borangle wearing its hat" without additional textual descriptions of its hat. We call this kind of generation textitpersonalized attribute-reasoning generation. To address the limitation, we present UniCTokens, a novel framework that effectively integrates personalized information into a unified vision language model (VLM) for understanding and generation. UniCTokens trains a set of unified concept tokens to leverage complementary semantics, boosting two personalized tasks. Moreover, we propose a progressive training strategy with three stages: understanding warm-up, bootstrapping generation from understanding, and deepening understanding from generation to enhance mutual benefits between both tasks. To quantitatively evaluate the unified VLM personalization, we present UnifyBench, the first benchmark for assessing concept understanding, concept generation, and attribute-reasoning generation. Experimental results on UnifyBench indicate that UniCTokens shows competitive performance compared to leading methods in concept understanding, concept generation, and achieving state-of-the-art results in personalized attribute-reasoning generation. Our research demonstrates that enhanced understanding improves generation, and the generation process can yield valuable insights into understanding. Our code and dataset will be released at: hrefhttps://github.com/arctanxarc/UniCTokenshttps://github.com/arctanxarc/UniCTokens.