mlsanityData to 7 Oct 2026

Mind-Omni: A Unified Multi-Task Framework for Brain-Vision-Language Modeling via Discrete Diffusion

arXiv 2605.29591 ↗First posted 28 May 2026cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Modeling the interplay between external stimuli and internal neural representations is a pivotal research area for Brain-Computer Interfaces (BCIs). A major limitation of prior work is the prevailing paradigm of specialized, single-task models, which curtails versatility and neglects inter-task synergies. To address this, we propose Mind-Omni, the first versatile framework that unifies seven distinct encoding and decoding tasks through a discrete diffusion paradigm. At its core is a novel Brain Tokenizer that transforms heterogeneous, continuous brain signals into standardized, discrete tokens. This enables direct, token-level interactions for mutual understanding and generation between any two or more modalities within a shared semantic space. To unlock advanced reasoning capabilities, we further curate a specialized Brain Question Answering (BQA) instruction-tuning dataset. Our model not only establishes a new state-of-the-art among multi-task unified frameworks but also provides strong evidence for multi-task synergy. By demonstrating performance competitive with, and at times superior to, larger specialized models, our work offers a powerful new paradigm for neural modeling and paves the way for foundation models of neural activity. The code is publicly available at https://github.com/ReedOnePeck/Mind-Omni.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 15 of its 100 references that are in the map. First posted on arXiv 28 May 2026.

Builds on

15 of 100 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  2. 2023OneLLM: One Framework to Align All Modalities with Language6 Dec

Other references · 13

  1. 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 960
  2. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  3. 2025Large Language Diffusion Modelscited by 872
  4. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  5. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  6. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  7. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468
  8. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  9. 2025Diffusion Transformers with Representation Autoencoderscited by 275
  10. 2023Decoding Natural Images from EEG for Object Recognitioncited by 79
  11. 2024Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusioncited by 79
  12. 2025Muddit: Liberating Generation Beyond Text-to-Image with a Unified Discrete Diffusion Modelcited by 52
1 more
  1. 2023Brain Captioning: Decoding human brain activity into images and textcited by 33

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Decoding the Multimodal Mind: Generalizable Brain-to-Text Translation via Multimodal Alignment and Adaptive Routing
  2. 2026The Platonic brain bridge hypothesis: human brain networks as an architectural prior for multimodal large language models
  3. 2025Uncovering Brain-Like Hierarchical Patterns in Vision-Language Models through fMRI-Based Neural Encoding
  4. 2024MindSemantix: Deciphering Brain Visual Experiences with a Brain-Language Model
  5. 2022Semantic Brain Decoding: from fMRI to conceptually similar image reconstruction of visual stimuli
  6. 2025Talking to the brain: Using Large Language Models as Proxies to Model Brain Semantic Representation
  7. 2022Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding
  8. 2026The Capacity of Thought: Benchmarking Llama 3.2 in Semantic fMRI Neural Language Decoding and Improving the Huth Encoding-Model Baseline