Data to 7 Oct 2026

Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization

arXiv 2403.08730 ↗First posted 13 Mar 2024cs.CL cs.CVcited by 75

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 2 more

LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment
Synthetic data and self-improvement for LLMs → Self-improving MLLMs and synthetic data

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) excel in generating responses based on visual inputs. However, they often suffer from a bias towards generating responses similar to their pretraining corpus, overshadowing the importance of visual information. We treat this bias as a "preference" for pretraining statistics, which hinders the model's grounding in visual input. To mitigate this issue, we propose Bootstrapped Preference Optimization (BPO), which conducts preference learning with datasets containing negative responses bootstrapped from the model itself. Specifically, we propose the following two strategies: 1) using distorted image inputs to the MLLM for eliciting responses that contain signified pretraining bias; 2) leveraging text-based LLM to explicitly inject erroneous but common elements into the original response. Those undesirable responses are paired with original annotated responses from the datasets to construct the preference dataset, which is subsequently utilized to perform preference learning. Our approach effectively suppresses pretrained LLM bias, enabling enhanced grounding in visual inputs. Extensive experimentation demonstrates significant performance improvements across multiple benchmarks, advancing the state-of-the-art in multimodal conversational systems.

Key facts. Cited by 75 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 87 references that are in the map. First posted on arXiv 13 Mar 2024.

Builds on

31 of 87 references.

References · 31

2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
2022Training Compute-Optimal Large Language Modelscited by 2,927
2019Fine-Tuning Language Models from Human Preferencescited by 2,124
2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
2023Prompt Injection attack against LLM-integrated Applicationscited by 568
2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
2022Improving alignment of dialogue agents via targeted human judgementscited by 527

19 more

2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
2018Scalable agent alignment via reward modeling: a research directioncited by 438
2023Aligning Text-to-Image Models using Human Feedbackcited by 418
2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
2021Recursively Summarizing Books with Human Feedbackcited by 287
2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preferencecited by 284
2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 237
2022Optimizing Prompts for Text-to-Image Generationcited by 204
2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
2023Mitigating the Alignment Tax of RLHFcited by 181
2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
2023Training Language Models with Language Feedback at Scalecited by 125
2019On the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 82
2022In conversation with Artificial Intelligence: aligning language models with human valuescited by 61
2023LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Modelscited by 58
2023HiLM-D: Enhancing MLLMs with Multi-Scale High-Resolution Details for Autonomous Drivingcited by 48

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 26; 2025: 39; 2026: 10.

75 papers.

Sort:

By area:

2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 132
2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 129
2024VideoDPO: Omni-Preference Alignment for Video Diffusion Generationcited by 102
2024mDPO: Conditional Preference Optimization for Multimodal Large Language Modelscited by 92
2024SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Modelcited by 82
2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 73
2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
2024Image Textualization: An Automatic Framework for Creating Accurate and Detailed Image Descriptionscited by 43
2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applicationscited by 40
2025MLLMs are Deeply Affected by Modality Biascited by 38

38 more

2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
2024Looking Beyond Text: Reducing Language bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidancecited by 32
2024CoCA: Regaining Safety-awareness of Multimodal Large Language Models with Constitutional Calibrationcited by 30
2025Temporal Preference Optimization for Long-Form Video Understandingcited by 28
2024Modality-Fair Preference Optimization for Trustworthy MLLM Alignmentcited by 26
2024Personalized Visual Instruction Tuningcited by 24
2024Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Modelscited by 23
2024Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimizationcited by 20
2025AVTrustBench: Assessing and Enhancing Reliability and Robustness in Audio-Visual LLMscited by 19
2024AlignGuard: Scalable Safety Alignment for Text-to-Image Generationcited by 17
2024TLDR: Token-Level Detective Reward Model for Large Vision Language Modelscited by 15
2025PaMi-VDPO: Mitigating Video Hallucinations by Prompt-Aware Multi-Instance Video Preference Learningcited by 15
2025Co-Reinforcement Learning for Unified Multimodal Understanding and Generationcited by 15
2025VistaDPO: Video Hierarchical Spatial-Temporal Direct Preference Optimization for Large Video Modelscited by 14
2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
2025Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimizationcited by 14
2025Analyzing and Mitigating Object Hallucination: A Training Bias Perspectivecited by 12
2025MCA-LLaVA: Manhattan Causal Attention for Reducing Hallucination in Large Vision-Language Modelscited by 11
2026On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMscited by 11
2024OPTune: Efficient Online Preference Tuningcited by 10
2024SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimizationcited by 10
2024NoiseBoost: Alleviating Hallucination with Noise Perturbation for Multimodal Large Language Modelscited by 9
2024Medical MLLM is Vulnerable: Cross-Modality Jailbreak and Mismatched Attacks on Medical Multimodal Large Language Modelscited by 8
2024Mitigating Hallucination in Visual-Language Models via Re-Balancing Contrastive Decodingcited by 8
2025VideoPASTA: 7K Preference Pairs That Matter for Video-LLM Alignmentcited by 8
2025Modality-Balancing Preference Optimization of Large Multimodal Models by Adversarial Negative Miningcited by 8
2025Multi-speaker Attention Alignment for Multimodal Social Interactioncited by 8
2025MR. Judge: Multimodal Reasoner as a Judgecited by 7
2025Mol-LLM: Multimodal Generalist Molecular LLM with Improved Graph Utilizationcited by 6
2025Decoupling Contrastive Decoding: Robust Hallucination Mitigation in Multimodal Large Language Modelscited by 5
2025LPOI: Listwise Preference Optimization for Vision Language Modelscited by 5
2025NoisyGRPO: Incentivizing Multimodal CoT Reasoning via Noise Injection and Bayesian Estimationcited by 5
2026Visual Preference Optimization with Rubric Rewardscited by 5
2024Continual SFT Matches Multimodal RLHF with Negative Supervisioncited by 4
2025LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learningcited by 4
2025PositionIC: Unified Position and Identity Consistency for Image Customizationcited by 4
2025Pointing to a Llama and Call it a Camel: On the Sycophancy of Multimodal Large Language Modelscited by 4
2025Optimizing LVLMs with On-Policy Data for Effective Hallucination Mitigationcited by 4

50 most cited of 75 All 75 (JSON)

Similar, not cited

Close in wording, no citation either way