mlsanityData to 7 Oct 2026

DAMA: Data- and Model-aware Alignment of Multi-modal LLMs

arXiv 2502.01943 ↗First posted 4 Feb 2025cs.CVcited by 14

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Direct Preference Optimization (DPO) has shown effectiveness in aligning multi-modal large language models (MLLM) with human preferences. However, existing methods exhibit an imbalanced responsiveness to the data of varying hardness, tending to overfit on the easy-to-distinguish data while underfitting on the hard-to-distinguish data. In this paper, we propose Data- and Model-aware DPO (DAMA) to dynamically adjust the optimization process from two key aspects: (1) a data-aware strategy that incorporates data hardness, and (2) a model-aware strategy that integrates real-time model responses. By combining the two strategies, DAMA enables the model to effectively adapt to data with varying levels of hardness. Extensive experiments on five benchmarks demonstrate that DAMA not only significantly enhances the trustworthiness, but also improves the effectiveness over general tasks. For instance, on the Object-HalBench, our DAMA-7B reduces response-level and mentioned-level hallucination by 90.0% and 95.3%, respectively, surpassing the performance of GPT-4V.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 18 of its 39 references that are in the map. First posted on arXiv 4 Feb 2025.

Builds on

18 of 39 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024mDPO: Conditional Preference Optimization for Multimodal Large Language Models17 Jun
  2. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug

Other references · 16

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  4. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  5. 2018Object Hallucination in Image Captioningcited by 661
  6. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  7. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 293
  8. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 256
  9. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimizationcited by 223
  10. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 197
  11. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 129
  12. 2024MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Enginecited by 111
4 more
  1. 2024VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignmentcited by 56
  2. 2024Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidancecited by 44
  3. 2024Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignmentcited by 17
  4. 2024Token Preference Optimization with Self-Calibrated Visual-Anchored Rewards for Hallucination Mitigationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 6.

14 papers.

Sort:

By area:

  1. 2025Thyme: Think Beyond Imagescited by 160
  2. 2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learningcited by 65
  3. 2025Inverse Reinforcement Learning with Dynamic Reward Scaling for LLM Alignmentcited by 20
  4. 2025Accelerating Diffusion Transformer via Gradient-Optimized Cachecited by 15
  5. 2025BaseReward: A Strong Baseline for Multimodal Reward Modelcited by 15
  6. 2025Aligning Multimodal LLM with Human Preference: A Surveycited by 12
  7. 2025Modality-Balancing Preference Optimization of Large Multimodal Models by Adversarial Negative Miningcited by 8
  8. 2025AdaViP: Aligning Multi-modal LLMs via Adaptive Vision-enhanced Preference Optimizationcited by 6
  9. 2026Thinking with Frames: Generative Video Distortion Evaluation via Frame Reward Modelcited by 2
  10. 2026Beyond Where to Look: Trajectory-Guided Reinforcement Learning for Multimodal RLVRcited by 1
  11. 2026Multi2: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environmentscited by 1
  12. 2026Schr"oMind: Mitigating Hallucinations in Multimodal Large Language Models via Solving the Schr"odinger Bridge Problemcited by 0
2 more
  1. 2026Scalpel: Fine-Grained Alignment of Attention Activation Manifolds via Mixture Gaussian Bridges to Mitigate Multimodal Hallucinationcited by 0
  2. 2026CHILLGuard: Towards Fine-Grained Chinese LLM Safety Guardrail with Scalable Data Construction and Model-aware Preference Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Less is More: Improving LLM Alignment via Preference Data Selection
  2. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMs
  3. 2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimization
  4. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  5. 2025Data Selection for LLM Alignment Using Fine-Grained Preferences
  6. 2025ComPO: Preference Alignment via Comparison Oracles
  7. 2024Direct Large Language Model Alignment Through Self-Rewarding Contrastive Prompt Distillation
  8. 2025Direct Density Ratio Optimization: A Statistically Consistent Approach to Aligning Large Language Models