Mind-Omni: A Unified Multi-Task Framework for Brain-Vision-Language Modeling via Discrete Diffusion
On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning
also in 1 more
Abstract (arXiv)
Modeling the interplay between external stimuli and internal neural representations is a pivotal research area for Brain-Computer Interfaces (BCIs). A major limitation of prior work is the prevailing paradigm of specialized, single-task models, which curtails versatility and neglects inter-task synergies. To address this, we propose Mind-Omni, the first versatile framework that unifies seven distinct encoding and decoding tasks through a discrete diffusion paradigm. At its core is a novel Brain Tokenizer that transforms heterogeneous, continuous brain signals into standardized, discrete tokens. This enables direct, token-level interactions for mutual understanding and generation between any two or more modalities within a shared semantic space. To unlock advanced reasoning capabilities, we further curate a specialized Brain Question Answering (BQA) instruction-tuning dataset. Our model not only establishes a new state-of-the-art among multi-task unified frameworks but also provides strong evidence for multi-task synergy. By demonstrating performance competitive with, and at times superior to, larger specialized models, our work offers a powerful new paradigm for neural modeling and paves the way for foundation models of neural activity. The code is publicly available at https://github.com/ReedOnePeck/Mind-Omni.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 15 of its 100 references that are in the map. First posted on arXiv 28 May 2026.
Builds on
15 of 100 references.
Closest ancestors · 2
Same direction, newest first
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
- 2023OneLLM: One Framework to Align All Modalities with Language6 Dec
Other references · 13
- 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 960
- 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
- 2025Large Language Diffusion Modelscited by 872
- 2024Emu3: Next-Token Prediction is All You Needcited by 761
- 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
- 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
- 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468
- 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
- 2025Diffusion Transformers with Representation Autoencoderscited by 275
- 2023Decoding Natural Images from EEG for Object Recognitioncited by 79
- 2024Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusioncited by 79
- 2025Muddit: Liberating Generation Beyond Text-to-Image with a Unified Discrete Diffusion Modelcited by 52
1 more
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Decoding the Multimodal Mind: Generalizable Brain-to-Text Translation via Multimodal Alignment and Adaptive Routing
- 2026The Platonic brain bridge hypothesis: human brain networks as an architectural prior for multimodal large language models
- 2025Uncovering Brain-Like Hierarchical Patterns in Vision-Language Models through fMRI-Based Neural Encoding
- 2024MindSemantix: Deciphering Brain Visual Experiences with a Brain-Language Model
- 2022Semantic Brain Decoding: from fMRI to conceptually similar image reconstruction of visual stimuli
- 2025Talking to the brain: Using Large Language Models as Proxies to Model Brain Semantic Representation
- 2022Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding
- 2026The Capacity of Thought: Benchmarking Llama 3.2 in Semantic fMRI Neural Language Decoding and Improving the Huth Encoding-Model Baseline