# Enhancing Multimodal Unified Representations for Cross Modal Generalization

arXiv: https://arxiv.org/abs/2403.05168 · HTML: https://mlsanity.com/paper/2403.05168/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Mar 2024.

## Built on by (most cited first, 12 of 12)

- [WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling](https://mlsanity.com/paper/2408.16532/) (2024)
- [SpatialReward: Verifiable Spatial Reward Modeling for Fine-Grained Spatial Consistency in Text-to-Image Generation](https://mlsanity.com/paper/2603.22228/) (2026)
- [Semantic Residual for Multimodal Unified Discrete Representation](https://mlsanity.com/paper/2412.19128/) (2024)
- [Open-set Cross Modal Generalization via Multimodal Unified Representation](https://mlsanity.com/paper/2507.14935/) (2025)
- [TAP: Parameter-efficient Task-Aware Prompting for Adverse Weather Removal](https://mlsanity.com/paper/2508.07878/) (2025)
- [Enhancing Temporal Awareness in LLMs for Temporal Point Processes](https://mlsanity.com/paper/2601.00845/) (2025)
- [Continual Cross-Modal Generalization](https://mlsanity.com/paper/2504.00561/) (2025)
- [Asymmetric Hierarchical Anchoring for Robust Audio-Visual Cross-Modal Generalization](https://mlsanity.com/paper/2602.03570/) (2026)
- [Cross-Modal-Domain Generalization Through Semantically Aligned Discrete Representations](https://mlsanity.com/paper/2605.12145/) (2026)
- [CodeBind: Decoupled Representation Learning for Multimodal Alignment with Unified Compositional Codebook](https://mlsanity.com/paper/2605.18257/) (2026)
- [OmniVAE: An Audio-Video VAE with Cross-Modal Alignment for Joint Generation](https://mlsanity.com/paper/2607.23855/) (2026)
- [Codebook-Guided Cross-Modal Knowledge Distillation for Structurally Heterogeneous Features](https://mlsanity.com/paper/2609.37243/) (2026)

## Abstract (arXiv)

To enhance the interpretability of multimodal unified representations, many studies have focused on discrete unified representations. These efforts typically start with contrastive learning and gradually extend to the disentanglement of modal information, achieving solid multimodal discrete unified representations. However, existing research often overlooks two critical issues: 1) The use of Euclidean distance for quantization in discrete representations often overlooks the important distinctions among different dimensions of features, resulting in redundant representations after quantization; 2) Different modalities have unique characteristics, and a uniform alignment approach does not fully exploit these traits. To address these issues, we propose Training-free Optimization of Codebook (TOC) and Fine and Coarse cross-modal Information Disentangling (FCID). These methods refine the unified discrete representations from pretraining and perform fine- and coarse-grained information disentanglement tailored to the specific characteristics of each modality, achieving significant performance improvements over previous state-of-the-art models. The code is available at https://github.com/haihuangcode/CMG.