# OmniTaskonomy: When Does Visual Generation Improve Visual Understanding?

arXiv: https://arxiv.org/abs/2609.38079 · HTML: https://mlsanity.com/paper/2609.38079/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 71 references that are in the map.
- First posted on arXiv 29 Sep 2026.

## Builds on (most cited first, 21 of 21)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://mlsanity.com/paper/2405.09818/) (2024)
- [Emerging Properties in Unified Multimodal Pretraining](https://mlsanity.com/paper/2505.14683/) (2025)
- [BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset](https://mlsanity.com/paper/2505.09568/) (2025)
- [Modeling Context Between Objects for Referring Expression Understanding](https://mlsanity.com/paper/1608.00525/) (2016)
- [Transfer between Modalities with MetaQueries](https://mlsanity.com/paper/2504.06256/) (2025)
- [Cosmos 3: Omnimodal World Models for Physical AI](https://mlsanity.com/paper/2606.02800/) (2026)
- [BLIP3o-NEXT: Next Frontier of Native Image Generation](https://mlsanity.com/paper/2510.15857/) (2025)
- [Image Generators are Generalist Vision Learners](https://mlsanity.com/paper/2604.20329/) (2026)
- [SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecture](https://mlsanity.com/paper/2605.12500/) (2026)
- [UniEval: Unified Holistic Evaluation for Unified Multimodal Understanding and Generation](https://mlsanity.com/paper/2505.10483/) (2025)
- [Cross-Task Generalization Between Understanding and Generation in Unified Vision-Language Models: A Controlled Study](https://mlsanity.com/paper/2505.23043/) (2025)
- [Quantifying the Gap between Understanding and Generation within Unified Multimodal Models](https://mlsanity.com/paper/2602.02140/) (2026)
- [VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents](https://mlsanity.com/paper/2601.16973/) (2026)
- [Does Understanding Inform Generation in Unified Multimodal Models? From Analysis to Path Forward](https://mlsanity.com/paper/2511.20561/) (2025)
- [UniG2U-Bench: Do Unified Models Advance Multimodal Understanding?](https://mlsanity.com/paper/2603.03241/) (2026)
- [Uni-Edit: Intelligent Editing Is A General Task For Unified Model Tuning](https://mlsanity.com/paper/2605.21487/) (2026)
- [Beyond Accuracy: Benchmarking Cross-Task Consistency in Unified Multimodal Models](https://mlsanity.com/paper/2604.25072/) (2026)
- [UniHetero: Could Generation Enhance Understanding for Vision-Language-Model at Large Data Scale?](https://mlsanity.com/paper/2512.23512/) (2025)
- [Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes](https://mlsanity.com/paper/2608.05000/) (2026)
- [Unlocking the Potential of Image Editing via Concept Scaling and Dense Supervision](https://mlsanity.com/paper/2608.16812/) (2026)

## Abstract (arXiv)

Training a model to generate visual content can encourage it to learn rich perceptual capabilities related to geometry, spatial relationships, and objectness; yet, its benefits for visual understanding remain unclear. We ask: when and how does visual generation supervision improve visual understanding? We study controlled pairs of image-to-image (I2I) generation and image-to-text (I2T) understanding tasks that express the same underlying problem in different output modalities. We find that under the correct recipe, I2I training improves downstream I2T performance, with larger gains as the amount of I2I training data increases. We next ask which generation tasks benefit which understanding capabilities. To study transfer beyond paired tasks, we introduce OmniTaskonomy, a unified taxonomy spanning 19 I2I generation tasks and 25 I2T understanding capabilities. The resulting transfer map reveals selective, task-dependent benefits. Some follow intuitive correspondences, e.g., depth prediction improving metric 3D reasoning, object pointing improving counting, and jigsaw reconstruction improving 2D ordering. Interestingly, we also uncover surprising connections: 2.5D segmentation improving category recognition and Z-depth prediction improving localization. To probe these patterns, we analyze gradient alignment between generation and understanding tasks and find that stronger alignment is associated with larger downstream transfer gains. Together, our results highlight visual generation as a rich source of supervision for visual understanding and provide a roadmap for unlocking its benefits through the right training curriculum and task selection. Project page: https://omni-taskonomy.github.io/.