# Asymmetric Idiosyncrasies in Multimodal Models

arXiv: https://arxiv.org/abs/2602.22734 · HTML: https://mlsanity.com/paper/2602.22734/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 37 references that are in the map.
- First posted on arXiv 26 Feb 2026.

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Qwen-Image Technical Report](https://mlsanity.com/paper/2508.02324/) (2025)
- [On the detection of synthetic images generated by diffusion models](https://mlsanity.com/paper/2211.00680/) (2022)
- [Playground v3: Improving Text-to-Image Alignment with Deep-Fusion Large Language Models](https://mlsanity.com/paper/2409.10695/) (2024)
- [SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training?](https://mlsanity.com/paper/2402.01832/) (2024)
- [A Decade's Battle on Dataset Bias: Are We There Yet?](https://mlsanity.com/paper/2403.08632/) (2024)
- [Your Large Language Models Are Leaving Fingerprints](https://mlsanity.com/paper/2405.14057/) (2024)
- [Idiosyncrasies in Large Language Models](https://mlsanity.com/paper/2502.12150/) (2025)
- [From Text to Source: Results in Detecting Large Language Model-Generated Content](https://mlsanity.com/paper/2309.13322/) (2023)
- [VibeCheck: Discover and Quantify Qualitative Differences in Large Language Models](https://mlsanity.com/paper/2410.12851/) (2024)
- [Revisit Large-Scale Image-Caption Data in Pre-training Multimodal Foundation Models](https://mlsanity.com/paper/2410.02740/) (2024)
- [Are Images Indistinguishable to Humans Also Indistinguishable to Classifiers?](https://mlsanity.com/paper/2405.18029/) (2024)
- [How to Train your Text-to-Image Model: Evaluating Design Choices for Synthetic Training Captions](https://mlsanity.com/paper/2506.16679/) (2025)
- [Enhancing Fingerprint Image Synthesis with GANs, Diffusion Models, and Style Transfer Techniques](https://mlsanity.com/paper/2403.13916/) (2024)

## Built on by (most cited first, 1 of 1)

- [Do Text Edits Generalize to Visual Generation? Benchmarking Cross-Modal Knowledge Editing in UMMs](https://mlsanity.com/paper/2606.00477/) (2026)

## Abstract (arXiv)

In this work, we study idiosyncrasies in the caption models and their downstream impact on text-to-image models. We design a systematic analysis: given either a generated caption or the corresponding image, we train neural networks to predict the originating caption model. Our results show that text classification yields very high accuracy (99.70%), indicating that captioning models embed distinctive stylistic signatures. In contrast, these signatures largely disappear in the generated images, with classification accuracy dropping to at most 50% even for the state-of-the-art Flux model. To better understand this cross-modal discrepancy, we further analyze the data and find that the generated images fail to preserve key variations present in captions, such as differences in the level of detail, emphasis on color and texture, and the distribution of objects within a scene. Overall, our classification-based framework provides a novel methodology for quantifying both the stylistic idiosyncrasies of caption models and the prompt-following ability of text-to-image systems.