# Enhance Vision-Language Alignment with Noise

arXiv: https://arxiv.org/abs/2412.10817 · HTML: https://mlsanity.com/paper/2412.10817/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 39 references that are in the map.
- First posted on arXiv 14 Dec 2024.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)
- [VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks](https://mlsanity.com/paper/2112.06825/) (2021)
- [Uncertainty Modeling for Out-of-Distribution Generalization](https://mlsanity.com/paper/2202.03958/) (2022)
- [CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attention](https://mlsanity.com/paper/2209.14169/) (2022)
- [Adversarial Style Augmentation for Domain Generalized Urban-Scene Segmentation](https://mlsanity.com/paper/2207.04892/) (2022)
- [Variational Positive-incentive Noise: How Noise Benefits Models](https://mlsanity.com/paper/2306.07651/) (2023)
- [Data Augmentation of Contrastive Learning is Estimating Positive-incentive Noise](https://mlsanity.com/paper/2408.09929/) (2024)
- [Adversarial Style Augmentation for Domain Generalization](https://mlsanity.com/paper/2301.12643/) (2023)
- [LAMM: Label Alignment for Multi-Modal Prompt Learning](https://mlsanity.com/paper/2312.08212/) (2023)
- [Decouple Graph Neural Networks: Train Multiple Simple GNNs Simultaneously Instead of One](https://mlsanity.com/paper/2304.10126/) (2023)

## Built on by (most cited first, 30 of 34)

- [Learn Beneficial Noise as Graph Augmentation](https://mlsanity.com/paper/2505.19024/) (2025)
- [NFIG: Multi-Scale Autoregressive Image Generation via Frequency Ordering](https://mlsanity.com/paper/2503.07076/) (2025)
- [Adv-CPG: A Customized Portrait Generation Framework with Facial Adversarial Attacks](https://mlsanity.com/paper/2503.08269/) (2025)
- [Mixture of Noise for Pre-Trained Model-Based Class-Incremental Learning](https://mlsanity.com/paper/2509.16738/) (2025)
- [Object-AVEdit: An Object-level Audio-Visual Editing Model](https://mlsanity.com/paper/2510.00050/) (2025)
- [Integrating Reinforcement Learning with Visual Generative Models: Foundations and Advances](https://mlsanity.com/paper/2508.10316/) (2025)
- [Explore How to Inject Beneficial Noise in MLLMs](https://mlsanity.com/paper/2511.12917/) (2025)
- [Rectified Noise: A Generative Model Using Positive-incentive Noise](https://mlsanity.com/paper/2511.07911/) (2025)
- [MR-UIE: Multi-Perspective Reasoning with Reinforcement Learning for Universal Information Extraction](https://mlsanity.com/paper/2509.09082/) (2025)
- [Laytrol: Preserving Pretrained Knowledge in Layout Control for Multimodal Diffusion Transformers](https://mlsanity.com/paper/2511.07934/) (2025)
- [AnchorFormer: Differentiable Anchor Attention for Efficient Vision Transformer](https://mlsanity.com/paper/2505.16463/) (2025)
- [TableZoomer: A Collaborative Agent Framework for Large-scale Table Question Answering](https://mlsanity.com/paper/2509.01312/) (2025)
- [Enhancing Neural Video Compression of Static Scenes with Positive-Incentive Noise](https://mlsanity.com/paper/2603.06095/) (2026)
- [Learning with Dual-level Noisy Correspondence for Multi-modal Entity Alignment](https://mlsanity.com/paper/2510.18240/) (2025)
- [Class-Aware Prototype Learning with Negative Contrast for Test-Time Adaptation of Vision-Language Models](https://mlsanity.com/paper/2510.19802/) (2025)
- [GRPO-RM: Fine-Tuning Representation Models via GRPO-Driven Reinforcement Learning](https://mlsanity.com/paper/2511.15256/) (2025)
- [Perturbation-Regularized Open-Vocabulary Remote Sensing Segmentation with Unified Multi-Domain Evaluation](https://mlsanity.com/paper/2604.15652/) (2026)
- [Noise Augmented Fine Tuning for Mitigating Hallucinations in Large Language Models](https://mlsanity.com/paper/2504.03302/) (2025)
- [Unveiling Hidden Vulnerabilities in Digital Human Generation via Adversarial Attacks](https://mlsanity.com/paper/2504.17457/) (2025)
- [ERF-BA-TFD+: A Multimodal Model for Audio-Visual Deepfake Detection](https://mlsanity.com/paper/2508.17282/) (2025)
- [A Benchmark for Multi-Lingual Vision-Language Learning in Remote Sensing Image Captioning](https://mlsanity.com/paper/2503.04592/) (2025)
- [Real-Time Crowd Counting for Embedded Systems with Lightweight Architecture](https://mlsanity.com/paper/2510.13250/) (2025)
- [CoLM: Collaborative Large Models via A Client-Server Paradigm](https://mlsanity.com/paper/2511.06991/) (2025)
- [How Noise Benefits AI-generated Image Detection](https://mlsanity.com/paper/2511.16136/) (2025)
- [Multi-Intent Spoken Language Understanding: Methods, Trends, and Challenges](https://mlsanity.com/paper/2512.11258/) (2025)
- [Noise-Robust Tiny Object Localization with Flows](https://mlsanity.com/paper/2601.00617/) (2026)
- [Batch Loss Score for Dynamic Data Pruning](https://mlsanity.com/paper/2604.04681/) (2026)
- [[CLS] is Not Enough: Multi-Label Recognition via Patch-Level Inference and Adaptive Aggregation](https://mlsanity.com/paper/2605.25821/) (2026)
- [Semi-Supervised Noise Adaptation: Transferring Knowledge from Noise Domain](https://mlsanity.com/paper/2606.00558/) (2026)
- [Geometry-Preserving Unsupervised Alignment for Heterogeneous Foundation Models](https://mlsanity.com/paper/2606.04385/) (2026)

## Abstract (arXiv)

With the advancement of pre-trained vision-language (VL) models, enhancing the alignment between visual and linguistic modalities in downstream tasks has emerged as a critical challenge. Different from existing fine-tuning methods that add extra modules to these two modalities, we investigate whether the frozen model can be fine-tuned by customized noise. Our approach is motivated by the scientific study of beneficial noise, namely Positive-incentive Noise (Pi-noise or π-noise) , which quantitatively analyzes the impact of noise. It therefore implies a new scheme to learn beneficial noise distribution that can be employed to fine-tune VL models. Focusing on few-shot classification tasks based on CLIP, we reformulate the inference process of CLIP and apply variational inference, demonstrating how to generate π-noise towards visual and linguistic modalities. Then, we propose Positive-incentive Noise Injector (PiNI), which can fine-tune CLIP via injecting noise into both visual and text encoders. Since the proposed method can learn the distribution of beneficial noise, we can obtain more diverse embeddings of vision and language to better align these two modalities for specific downstream tasks within limited computational resources. We evaluate different noise incorporation approaches and network architectures of PiNI. The evaluation across 11 datasets demonstrates its effectiveness.