# DAMA: Data- and Model-aware Alignment of Multi-modal LLMs

arXiv: https://arxiv.org/abs/2502.01943 · HTML: https://mlsanity.com/paper/2502.01943/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 18 of its 39 references that are in the map.
- First posted on arXiv 4 Feb 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)
- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 18 of 18)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2308.02490/) (2023)
- [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://mlsanity.com/paper/2405.14734/) (2024)
- [Object Hallucination in Image Captioning](https://mlsanity.com/paper/1809.02156/) (2018)
- [A Survey on Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2402.00253/) (2024)
- [AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation](https://mlsanity.com/paper/2311.07397/) (2023)
- [KTO: Model Alignment as Prospect Theoretic Optimization](https://mlsanity.com/paper/2402.01306/) (2024)
- [Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization](https://mlsanity.com/paper/2311.16839/) (2023)
- [Aligning Modalities in Vision Large Language Models via Preference Fine-tuning](https://mlsanity.com/paper/2402.11411/) (2024)
- [RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthiness](https://mlsanity.com/paper/2405.17220/) (2024)
- [MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Engine](https://mlsanity.com/paper/2407.08739/) (2024)
- [mDPO: Conditional Preference Optimization for Multimodal Large Language Models](https://mlsanity.com/paper/2406.11839/) (2024)
- [VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignment](https://mlsanity.com/paper/2410.09421/) (2024)
- [Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance](https://mlsanity.com/paper/2402.08680/) (2024)
- [Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment](https://mlsanity.com/paper/2410.14148/) (2024)
- [Token Preference Optimization with Self-Calibrated Visual-Anchored Rewards for Hallucination Mitigation](https://mlsanity.com/paper/2412.14487/) (2024)

## Built on by (most cited first, 14 of 14)

- [Thyme: Think Beyond Images](https://mlsanity.com/paper/2508.11630/) (2025)
- [R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learning](https://mlsanity.com/paper/2505.02835/) (2025)
- [Inverse Reinforcement Learning with Dynamic Reward Scaling for LLM Alignment](https://mlsanity.com/paper/2503.18991/) (2025)
- [Accelerating Diffusion Transformer via Gradient-Optimized Cache](https://mlsanity.com/paper/2503.05156/) (2025)
- [BaseReward: A Strong Baseline for Multimodal Reward Model](https://mlsanity.com/paper/2509.16127/) (2025)
- [Aligning Multimodal LLM with Human Preference: A Survey](https://mlsanity.com/paper/2503.14504/) (2025)
- [Modality-Balancing Preference Optimization of Large Multimodal Models by Adversarial Negative Mining](https://mlsanity.com/paper/2506.08022/) (2025)
- [AdaViP: Aligning Multi-modal LLMs via Adaptive Vision-enhanced Preference Optimization](https://mlsanity.com/paper/2504.15619/) (2025)
- [Thinking with Frames: Generative Video Distortion Evaluation via Frame Reward Model](https://mlsanity.com/paper/2601.04033/) (2026)
- [Multi2: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments](https://mlsanity.com/paper/2606.03698/) (2026)
- [Beyond Where to Look: Trajectory-Guided Reinforcement Learning for Multimodal RLVR](https://mlsanity.com/paper/2603.26126/) (2026)
- [Schr"oMind: Mitigating Hallucinations in Multimodal Large Language Models via Solving the Schr"odinger Bridge Problem](https://mlsanity.com/paper/2602.09528/) (2026)
- [Scalpel: Fine-Grained Alignment of Attention Activation Manifolds via Mixture Gaussian Bridges to Mitigate Multimodal Hallucination](https://mlsanity.com/paper/2602.09541/) (2026)
- [CHILLGuard: Towards Fine-Grained Chinese LLM Safety Guardrail with Scalable Data Construction and Model-aware Preference Alignment](https://mlsanity.com/paper/2606.15396/) (2026)

## Abstract (arXiv)

Direct Preference Optimization (DPO) has shown effectiveness in aligning multi-modal large language models (MLLM) with human preferences. However, existing methods exhibit an imbalanced responsiveness to the data of varying hardness, tending to overfit on the easy-to-distinguish data while underfitting on the hard-to-distinguish data. In this paper, we propose Data- and Model-aware DPO (DAMA) to dynamically adjust the optimization process from two key aspects: (1) a data-aware strategy that incorporates data hardness, and (2) a model-aware strategy that integrates real-time model responses. By combining the two strategies, DAMA enables the model to effectively adapt to data with varying levels of hardness. Extensive experiments on five benchmarks demonstrate that DAMA not only significantly enhances the trustworthiness, but also improves the effectiveness over general tasks. For instance, on the Object-HalBench, our DAMA-7B reduces response-level and mentioned-level hallucination by 90.0% and 95.3%, respectively, surpassing the performance of GPT-4V.