# Beyond Accuracy: Benchmarking Cross-Task Consistency in Unified Multimodal Models

arXiv: https://arxiv.org/abs/2604.25072 · HTML: https://mlsanity.com/paper/2604.25072/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 47 references that are in the map.
- First posted on arXiv 27 Apr 2026.

## Builds on (most cited first, 30 of 45)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](https://mlsanity.com/paper/2310.11513/) (2023)
- [SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](https://mlsanity.com/paper/2307.16125/) (2023)
- [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://mlsanity.com/paper/2405.09818/) (2024)
- [Emerging Properties in Unified Multimodal Pretraining](https://mlsanity.com/paper/2505.14683/) (2025)
- [Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling](https://mlsanity.com/paper/2501.17811/) (2025)
- [Emu3: Next-Token Prediction is All You Need](https://mlsanity.com/paper/2409.18869/) (2024)
- [Show-o: One Single Transformer to Unify Multimodal Understanding and Generation](https://mlsanity.com/paper/2408.12528/) (2024)
- [Scene Graph Generation by Iterative Message Passing](https://mlsanity.com/paper/1701.02426/) (2017)
- [Neural Motifs: Scene Graph Parsing with Global Context](https://mlsanity.com/paper/1711.06640/) (2017)
- [BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset](https://mlsanity.com/paper/2505.09568/) (2025)
- [MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers](https://mlsanity.com/paper/2012.00759/) (2020)
- [OneFormer: One Transformer to Rule Universal Image Segmentation](https://mlsanity.com/paper/2211.06220/) (2022)
- [T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generation](https://mlsanity.com/paper/2307.06350/) (2023)
- [MMaDA: Multimodal Large Diffusion Language Models](https://mlsanity.com/paper/2505.15809/) (2025)
- [Show-o2: Improved Native Unified Multimodal Models](https://mlsanity.com/paper/2506.15564/) (2025)
- [K-Net: Towards Unified Image Segmentation](https://mlsanity.com/paper/2106.14855/) (2021)
- [Effectively Unbiased FID and Inception Score and where to find them](https://mlsanity.com/paper/1911.07023/) (2019)
- [UniTok: A Unified Tokenizer for Visual Generation and Understanding](https://mlsanity.com/paper/2502.20321/) (2025)
- [A Comprehensive Survey of Scene Graphs: Generation and Application](https://mlsanity.com/paper/2104.01111/) (2021)

## Built on by (most cited first, 5 of 5)

- [TORUS: A Test of Rendering-Understanding Self-Coherence for Unified Audio Models](https://mlsanity.com/paper/2607.28896/) (2026)
- [Where a New Concept Must Enter: Entry Point Gates Cross-Task Usability in Unified Multimodal Models](https://mlsanity.com/paper/2608.17564/) (2026)
- [OmniTaskonomy: When Does Visual Generation Improve Visual Understanding?](https://mlsanity.com/paper/2609.38079/) (2026)
- [Does Gradient Conflict Predict the Understanding--Generation Trade-off? A Controlled Audit of Conflict-Metric Validity in Unified Multimodal Models](https://mlsanity.com/paper/2609.38465/) (2026)
- [Beyond Layers: Position-Resolved Gradient Conflict and Position-Aware Modulation for Unified Multimodal Models](https://mlsanity.com/paper/2609.38485/) (2026)

## Abstract (arXiv)

Unified Multimodal Models (uMMs) aim to support both visual understanding and visual generation within a shared representation. However, existing evaluation protocols assess these two capabilities independently and do not examine whether they are semantically aligned. As a result, it remains unclear whether current uMMs learn coherent unified representations that remain consistent across tasks given a visual concept. We introduce XTC-Bench, a scene-graph-grounded evaluation framework that measures cross-task visual semantic consistency. By deriving both generation prompts and understanding queries from a structured scene graph, our framework enables fact-level alignment analysis across objects, attributes, and relations. We propose Continuous Cross-Task Agreement (CCTA), a fine-grained metric that quantifies semantic agreement between generation and understanding over matched atomic facts, isolating internal consistency from standalone task accuracy. Extensive experiments on eight open-source and one commercial unified models reveal that high generation or understanding performance does not imply strong cross-task alignment, and architectural analysis shows consistency is governed by how tightly learning objectives are coupled across modalities, not by architectural unification alone. XTC-Bench provides a reproducible and model-agnostic framework for diagnosing representation-level misalignment, offering a concrete direction for advancing unified multimodal modeling beyond isolated task performance.