Gestalt: Large Multimodal Interplay Model
On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning also fits: Image editing and quality assessment
Abstract (arXiv)
In this paper, we propose Gestalt, a new paradigm of large multimodal model built around multimodal interplay. Despite rapid advances, large multimodal models are reaching a bottleneck: existing approaches focus primarily on accommodating additional modalities while overlooking the distinct characteristics of each modality and the relations among them. Motivated by the multistage property of human multisensory perception, we propose a multimodal interplay pyramid that organizes multimodal modeling as a progression from modality-specific processing, through cross-modal alignment, to deeper multimodal integration. Guided by this pyramid, Gestalt adopts a unified discrete diffusion framework and an interplay-partitioned architecture, with learnable interplay tokens mediating cross-modal exchange and integration. The pyramid also structures its data organization and training strategy. Strong performance across image generation, multimodal understanding, and text-only evaluation shows that Gestalt significantly improves cross-modal integration while preserving modality-specific information, effectively harnessing the strengths of diffusion-based multimodal models and offering a promising path toward unified multimodal intelligence. Project page: https://GeWu-Lab.github.io/Gestalt.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 47 references that are in the map. First posted on arXiv 30 Sep 2026.
Builds on
39 of 47 references.
Closest ancestors · 7
Same direction, newest first
- 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training28 Sep
- 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning22 May
- 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale6 Dec
- 2024Core Knowledge Deficits in Multi-Modal Language Models6 Oct
- 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
- 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs11 Jan
- 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
Other references · 32
- 2023Visual Instruction Tuningcited by 7,792
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
- 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
- 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
- 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
- 2025Large Language Diffusion Modelscited by 872
- 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
- 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 787
- 2024Emu3: Next-Token Prediction is All You Needcited by 761
- 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
- 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468
20 more
- 2025BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Datasetcited by 363
- 2017Gated Multimodal Units for Information Fusioncited by 244
- 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
- 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
- 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 131
- 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 130
- 2024SkySenseGPT: A Fine-Grained Instruction Tuning Dataset and Model for Remote Sensing Vision-Language Understandingcited by 101
- 2025Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decodingcited by 101
- 2026Kimi K3: Open Frontier Intelligencecited by 99
- 2025ShareGPT-4o-Image: Aligning Multimodal Models with GPT-4o-Level Image Generationcited by 94
- 2025Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understandingcited by 83
- 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
- 2025LaViDa: A Large Diffusion Language Model for Multimodal Understandingcited by 81
- 2025TIIF-Bench: How Does Your T2I Model Follow Your Instructions?cited by 64
- 2025FineVision: Open Data Is All You Needcited by 51
- 2024Scalable Image Tokenization with Index Backpropagation Quantizationcited by 44
- 2025UniGenBench++: A Unified Semantic Evaluation Benchmark for Text-to-Image Generationcited by 23
- 2026Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusioncited by 9
- 2026Advancing Aesthetic Image Generation via Composition Transfercited by 2
- 2026MIBench: Evaluating LMMs on Multimodal Interactioncited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2023Alternating Gradient Descent and Mixture-of-Experts for Integrated Multimodal Perception
- 2026Exploring Diffusion Transformers for Cross-Modal Augmentation in Multimodal Brain State Decoding
- 2024LLMBind: A Unified Modality-Task Integration Framework
- 2025VisiPruner: Decoding Discontinuous Cross-Modal Dynamics for Efficient Multimodal LLMs
- 2026All modalities are equal, but video is more equal: Closing the Cross-Attention Gap in Joint Video Generation
- 2026Toward Native Multimodal Modeling: A Roadmap
- 2023ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst
- 2026Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes