# MMRL: Multi-Modal Representation Learning for Vision-Language Models

arXiv: https://arxiv.org/abs/2503.08497 · HTML: https://mlsanity.com/paper/2503.08497/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 2 of its 57 references that are in the map.
- First posted on arXiv 11 Mar 2025.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 2 of 2)

- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [FILIP: Fine-grained Interactive Language-Image Pre-Training](https://mlsanity.com/paper/2111.07783/) (2021)

## Built on by (most cited first, 30 of 39)

- [Conditional Representation Learning for Customized Tasks](https://mlsanity.com/paper/2510.04564/) (2025)
- [MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptation](https://mlsanity.com/paper/2602.21397/) (2026)
- [MMRL++: Parameter-Efficient and Interaction-Aware Representation Learning for Vision-Language Models](https://mlsanity.com/paper/2505.10088/) (2025)
- [pFedMMA: Personalized Federated Fine-Tuning with Multi-Modal Adapter for Vision-Language Models](https://mlsanity.com/paper/2507.05394/) (2025)
- [Interpretable Cross-Domain Few-Shot Learning with Rectified Target-Domain Local Alignment](https://mlsanity.com/paper/2603.17655/) (2026)
- [Provably Improving Generalization of Few-Shot Models with Synthetic Data](https://mlsanity.com/paper/2505.24190/) (2025)
- [Raw Data Matters: Enhancing Prompt Tuning by Internal Augmentation on Vision-Language Models](https://mlsanity.com/paper/2508.02671/) (2025)
- [VaMP: Variational Multi-Modal Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2511.22664/) (2025)
- [Beyond Graph Model: Reliable VLM Fine-Tuning via Random Graph Adapter](https://mlsanity.com/paper/2507.10355/) (2025)
- [SemPT: Semantic Prompt Tuning for Vision-Language Models](https://mlsanity.com/paper/2508.10645/) (2025)
- [MetaTPT: Meta Test-time Prompt Tuning for Vision-Language Models](https://mlsanity.com/paper/2512.12268/) (2025)
- [DeAR: Fine-Grained VLM Adaptation by Decomposing Attention Head Roles](https://mlsanity.com/paper/2603.01111/) (2026)
- [FVG-PT: Adaptive Foreground View-Guided Prompt Tuning for Vision-Language Models](https://mlsanity.com/paper/2603.08708/) (2026)
- [AVION: Aerial Vision-Language Instruction from Offline Teacher to Prompt-Tuned Network](https://mlsanity.com/paper/2603.12659/) (2026)
- [ACE-LoRA: Graph-Attentive Context Enhancement for Parameter-Efficient Adaptation of Medical Vision-Language Models](https://mlsanity.com/paper/2603.17079/) (2026)
- [Variational Adapter for Cross-modal Similarity Representation](https://mlsanity.com/paper/2605.30968/) (2026)
- [ViP2-CLIP: Visual-Perception Prompting with Unified Alignment for Zero-Shot Anomaly Detection](https://mlsanity.com/paper/2505.17692/) (2025)
- [Adapting Foundation Vision-Language Models to Medical Diagnosis via Query-Driven Expert Bridging](https://mlsanity.com/paper/2505.21698/) (2025)
- [AGA: An adaptive group alignment framework for structured medical cross-modal representation learning](https://mlsanity.com/paper/2507.23402/) (2025)
- [Preserve and Sculpt: Manifold-Aligned Fine-tuning of Vision-Language Models for Few-Shot Learning](https://mlsanity.com/paper/2508.12877/) (2025)
- [ReBaPL: Repulsive Bayesian Prompt Learning](https://mlsanity.com/paper/2511.17339/) (2025)
- [YOLOA: Real-Time Affordance Detection via LLM Adapter](https://mlsanity.com/paper/2512.03418/) (2025)
- [Fourier-Attentive Representation Learning: A Fourier-Guided Framework for Few-Shot Generalization in Vision-Language Models](https://mlsanity.com/paper/2512.04395/) (2025)
- [AmPLe: Supporting Vision-Language Models via Adaptive-Debiased Ensemble Multi-Prompt Learning](https://mlsanity.com/paper/2512.18411/) (2025)
- [OmniVaT: Single Domain Generalization for Multimodal Visual-Tactile Learning](https://mlsanity.com/paper/2601.00352/) (2026)
- [MultiModal Fine-tuning with Synthetic Captions](https://mlsanity.com/paper/2601.21426/) (2026)
- [TADS: Task-Aware Data Selection for Multi-Task Multimodal Pre-Training](https://mlsanity.com/paper/2602.05251/) (2026)
- [Gram-Anchored Prompt Learning for Vision-Language Models via Second-Order Statistics](https://mlsanity.com/paper/2604.03980/) (2026)
- [Cluster-Aware Neural Collapse Prompt Tuning for Long-Tailed Generalization of Vision-Language Models](https://mlsanity.com/paper/2605.11939/) (2026)
- [A3B2: Adaptive Asymmetric Adapter for Alleviating Branch Bias in Vision-Language Image Classification with Few-Shot Learning](https://mlsanity.com/paper/2605.13161/) (2026)

## Abstract (arXiv)

Large-scale pre-trained Vision-Language Models (VLMs) have become essential for transfer learning across diverse tasks. However, adapting these models with limited few-shot data often leads to overfitting, diminishing their performance on new tasks. To tackle this issue, we propose a novel Multi-Modal Representation Learning (MMRL) framework that introduces a shared, learnable, and modality-agnostic representation space. MMRL projects the space tokens to text and image representation tokens, facilitating more effective multi-modal interactions. Unlike previous approaches that solely optimize class token features, MMRL integrates representation tokens at higher layers of the encoders--where dataset-specific features are more prominent--while preserving generalized knowledge in the lower layers. During training, both representation and class features are optimized, with trainable projection layer applied to the representation tokens, whereas the class token projection layer remains frozen to retain pre-trained knowledge. Furthermore, a regularization term is introduced to align the class features and text features with the zero-shot features from the frozen VLM, thereby safeguarding the model's generalization capacity. For inference, a decoupling strategy is employed, wherein both representation and class features are utilized for base classes, while only the class features, which retain more generalized knowledge, are used for new tasks. Extensive experiments across 15 datasets demonstrate that MMRL outperforms state-of-the-art methods, achieving a balanced trade-off between task-specific adaptation and generalization. Code is available at https://github.com/yunncheng/MMRL.