Lance: Unified Multimodal Modeling by Multi-Task Synergy
Not on the map. Why?
Abstract (arXiv)
We present Lance, a lightweight native unified model supporting multimodal understanding, generation, and editing for both images and videos. Rather than relying on model capacity scaling or text-image-dominant designs, Lance explores a practical paradigm for unified multimodal modeling via collaborative multi-task training. It is grounded in two core principles: unified context modeling and decoupled capability pathways. Specifically, Lance is trained from scratch and employs a dual-stream mixture-of-experts architecture on shared interleaved multimodal sequences, enabling joint context learning while decoupling the pathways for understanding and generation. We further introduce modality-aware rotary positional encoding to mitigate interference among heterogeneous visual tokens and boost cross-task alignment. During training, Lance adopts a staged multi-task training paradigm with capability-oriented objectives and adaptive data scheduling to strengthen both semantic comprehension and visual generation performance. Experimental results demonstrate that Lance substantially outperforms existing open-source unified models in image and video generation, while retaining strong multimodal understanding capabilities. The homepage is available at https://lance-project.github.io.
Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 134 of its 146 references that are in the map. First posted on arXiv 18 May 2026.
Builds on
134 of 146 references.
References · 134
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020Denoising Diffusion Probabilistic Modelscited by 21,076
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2023GPT-4 Technical Reportcited by 18,396
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2023Visual Instruction Tuningcited by 7,869
- 2022Scalable Diffusion Models with Transformerscited by 6,054
- 2022Classifier-Free Diffusion Guidancecited by 5,847
- 2025Qwen2.5-VL Technical Reportcited by 5,587
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
- 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
38 more
- 2021Zero-Shot Text-to-Image Generationcited by 4,288
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
- 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
- 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
- 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
- 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
- 2025Qwen3-VL Technical Reportcited by 2,793
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
- 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
- 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
- 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
- 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
- 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
- 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
- 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
- 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
- 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
- 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
- 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
- 2022MaskGIT: Masked Generative Image Transformercited by 1,099
- 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
- 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
- 2025Qwen-Image Technical Reportcited by 1,044
- 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
- 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 1,006
- 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
- 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
- 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
- 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
- 2023VideoChat: Chat-Centric Video Understandingcited by 857
- 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
- 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
- 2024Emu3: Next-Token Prediction is All You Needcited by 781
- 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
- 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 703
- 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
- 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
- 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 661
50 most cited of 134 All 134 (JSON)
Built on by
Citing papers per year: 2026: 12.
12 papers.
- 2026Toward Native Multimodal Modeling: A Roadmapcited by 7
- 2026Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipescited by 3
- 2026GRNEdit: Efficient General Video Editing from a New Binary-Evidence Perspective in Generative Refinement Networkscited by 2
- 2026Generation as Auxiliary Supervision: Enhancing Visual Understanding at Zero Inference Overhead via Decoupled Embedding Predictioncited by 1
- 2026Are Image Generators Zero-Shot Perceivers? A Rigorous Evaluationcited by 1
- 2026MachEmbodied-U0: Unified Understanding and Generation Model for Embodied Intelligencecited by 1
- 2026PixelUMM: Encoder-Free Unified Image and Video Understanding and Generationcited by 1
- 2026HarmVideoBench: Benchmarking Harmful Video Understanding in Large Multimodal Modelscited by 0
- 2026MoRoute: Dynamic Routing for In-Context Multimodal Video Generationcited by 0
- 2026Test-Time Curriculum for Open-Set AIGC Detectioncited by 0
- 2026HounsWorld: A Multimodal World Model for Hidden Patient-State Readout, Reconstruction, and Simulationcited by 0
- 2026Can Learning to Answer Spatial Questions Improve Novel-View Generation?cited by 0
- 2026Can Learning to Answer Spatial Questions Improve Novel-View Generation?14 Oct 2026
- 2026PixelUMM: Encoder-Free Unified Image and Video Understanding and Generation29 Sep 2026
- 2026MachEmbodied-U0: Unified Understanding and Generation Model for Embodied Intelligence22 Sep 2026
- 2026Are Image Generators Zero-Shot Perceivers? A Rigorous Evaluation7 Sep 2026
- 2026GRNEdit: Efficient General Video Editing from a New Binary-Evidence Perspective in Generative Refinement Networks17 Aug 2026
- 2026HounsWorld: A Multimodal World Model for Hidden Patient-State Readout, Reconstruction, and Simulation13 Aug 2026
- 2026Generation as Auxiliary Supervision: Enhancing Visual Understanding at Zero Inference Overhead via Decoupled Embedding Prediction12 Aug 2026
- 2026Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes5 Aug 2026
- 2026Test-Time Curriculum for Open-Set AIGC Detection1 Aug 2026
- 2026MoRoute: Dynamic Routing for In-Context Multimodal Video Generation31 Jul 2026
- 2026HarmVideoBench: Benchmarking Harmful Video Understanding in Large Multimodal Models25 Jun 2026
- 2026Toward Native Multimodal Modeling: A Roadmap25 May 2026
Similar, not cited
Close in wording, no citation either way
- 2025UniModel: A Visual-Only Framework for Unified Multimodal Understanding and Generation
- 2025ARMOR: Empowering Multimodal Understanding Model with Interleaved Multimodal Generation Capability
- 2026Omni-R1: Towards the Unified Generative Paradigm for Multimodal Reasoning
- 2025OmniMamba: Efficient and Unified Multimodal Understanding and Generation via State Space Models
- 2026DuoGen: Towards General Purpose Interleaved Multimodal Generation
- 2026Do Unified Multimodal Models Think in One Space? A Lens Through Cross-Branch Steering
- 2025ToFu: Visual Tokens Reduction via Fusion for Multi-modal, Multi-patch, Multi-image Task
- 2026Argus-Unified: Towards A Compact and Economical Unified Model for Image Understanding and Generation