# Mull-Tokens: Modality-Agnostic Latent Thinking

arXiv: https://arxiv.org/abs/2512.10941 · HTML: https://mlsanity.com/paper/2512.10941/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 57 of its 79 references that are in the map.
- First posted on arXiv 11 Dec 2025.

## Builds on (most cited first, 30 of 57)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://mlsanity.com/paper/2405.09818/) (2024)
- [Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs](https://mlsanity.com/paper/2406.16860/) (2024)
- [Training Large Language Models to Reason in a Continuous Latent Space](https://mlsanity.com/paper/2412.06769/) (2024)
- [Multimodal Chain-of-Thought Reasoning in Language Models](https://mlsanity.com/paper/2302.00923/) (2023)
- [Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces](https://mlsanity.com/paper/2412.14171/) (2024)
- [BLINK: Multimodal Large Language Models Can See but Not Perceive](https://mlsanity.com/paper/2404.12390/) (2024)
- [Video-R1: Reinforcing Video Reasoning in MLLMs](https://mlsanity.com/paper/2503.21776/) (2025)
- [Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation](https://mlsanity.com/paper/2410.13848/) (2024)
- [CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models](https://mlsanity.com/paper/2503.22020/) (2025)
- [Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach](https://mlsanity.com/paper/2502.05171/) (2025)
- [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://mlsanity.com/paper/2503.09567/) (2025)
- [BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset](https://mlsanity.com/paper/2505.09568/) (2025)
- [VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generation](https://mlsanity.com/paper/2409.04429/) (2024)
- [Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective](https://mlsanity.com/paper/2305.15408/) (2023)
- [Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models](https://mlsanity.com/paper/2406.09403/) (2024)
- [Think before you speak: Training Language Models With Pause Tokens](https://mlsanity.com/paper/2310.02226/) (2023)
- [Show-o2: Improved Native Unified Multimodal Models](https://mlsanity.com/paper/2506.15564/) (2025)
- [CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation](https://mlsanity.com/paper/2502.21074/) (2025)
- [Imagine while Reasoning in Space: Multimodal Visualization-of-Thought](https://mlsanity.com/paper/2501.07542/) (2025)
- [MolmoAct: Action Reasoning Models that can Reason in Space](https://mlsanity.com/paper/2508.07917/) (2025)
- [MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence](https://mlsanity.com/paper/2505.23764/) (2025)
- [Vision language models are blind: Failing to translate detailed visual features into words](https://mlsanity.com/paper/2407.06581/) (2024)

## Built on by (most cited first, 19 of 19)

- [The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlook](https://mlsanity.com/paper/2604.02029/) (2026)
- [LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Driving](https://mlsanity.com/paper/2603.01928/) (2026)
- [Forest Before Trees: Latent Superposition for Efficient Visual Reasoning](https://mlsanity.com/paper/2601.06803/) (2026)
- [CoWorld-VLA: Thinking in a Multi-Expert World Model for Autonomous Driving](https://mlsanity.com/paper/2605.10426/) (2026)
- [HyLaR: Hybrid Latent Reasoning with Decoupled Policy Optimization](https://mlsanity.com/paper/2604.20328/) (2026)
- [Semantic-Enriched Latent Visual Reasoning](https://mlsanity.com/paper/2605.19342/) (2026)
- [Do multimodal models imagine electric sheep?](https://mlsanity.com/paper/2605.09693/) (2026)
- [Ablate-to-Validate: Are Vision-Language Models Really Using Continuous Thought Tokens?](https://mlsanity.com/paper/2605.21642/) (2026)
- [STORM: Internalized Modeling for Spatial-Temporal Reasoning in Video-Language Models](https://mlsanity.com/paper/2605.26014/) (2026)
- [DeepLatent: Think with Images via Parallel Latent Visual Reasoning](https://mlsanity.com/paper/2606.00562/) (2026)
- [Imaginative Perception Tokens Enhance Spatial Reasoning in Multimodal Language Models](https://mlsanity.com/paper/2606.03988/) (2026)
- [GeoAnchor: Collaborative Reasoning via Latent Decomposition for 3D Spatial Understanding](https://mlsanity.com/paper/2607.13454/) (2026)
- [APIVOT: Adaptive Planning with Interleaved Vision-Language Thoughts](https://mlsanity.com/paper/2607.08024/) (2026)
- [Perception Before Reasoning: Dynamic Latent Reasoning for Video Understanding and Question Answering](https://mlsanity.com/paper/2608.04124/) (2026)
- [Uni-LaDiR: Latent Diffusion Unifies Multimodal Reasoning](https://mlsanity.com/paper/2609.19878/) (2026)
- [Next Thoughts Are Distributions: Generative Autoregressive Reasoning in the Latent Space](https://mlsanity.com/paper/2609.33271/) (2026)
- [Soft Spatial Reasoning](https://mlsanity.com/paper/2609.38717/) (2026)
- [SLVR: Structured Latent Visual Reasoning via Human-like Reasoning Flows](https://mlsanity.com/paper/2610.10563/) (2026)
- [Adaptive Look, Flexible Thought: Learning How and How Long to Think in Multimodal Reasoning](https://mlsanity.com/paper/2610.15810/) (2026)

## Abstract (arXiv)

Reasoning goes beyond language; the real world requires reasoning about space, time, affordances, and much more that words alone cannot convey. Existing multimodal models exploring the potential of reasoning with images are brittle and do not scale. They rely on calling specialist tools, costly generation of images, or handcrafted reasoning data to switch between text and image thoughts. Instead, we offer a simpler alternative -- Mull-Tokens -- modality-agnostic latent tokens pre-trained to hold intermediate information in either image or text modalities to let the model think free-form towards the correct answer. We investigate best practices to train Mull-Tokens inspired by latent reasoning frameworks. We first train Mull-Tokens using supervision from interleaved text-image traces, and then fine-tune without any supervision by only using the final answers. Across four challenging spatial reasoning benchmarks involving tasks such as solving puzzles and taking different perspectives, we demonstrate that Mull-Tokens improve upon several baselines utilizing text-only reasoning or interleaved image-text reasoning, achieving a +3% average improvement and up to +16% on a puzzle solving reasoning-heavy split compared to our strongest baseline. Adding to conversations around challenges in grounding textual and visual reasoning, Mull-Tokens offers a simple solution to abstractly think in multiple modalities.