mlsanityData to 14 Oct 2026

ASPO: Adaptive Sentence-Level Preference Optimization for Fine-Grained Multimodal Reasoning

arXiv 2505.19100 ↗First posted 25 May 2025cs.CL cs.CVcited by 2

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Direct Preference Optimization (DPO) has gained significant attention for its simplicity and computational efficiency in aligning large language models (LLMs). Recent advancements have extended DPO to multimodal scenarios, achieving strong performance. However, traditional DPO relies on binary preference optimization, rewarding or penalizing entire responses without considering fine-grained segment correctness, leading to suboptimal solutions. The root of this issue lies in the absence of fine-grained supervision during the optimization process. To address this, we propose Adaptive Sentence-level Preference Optimization (ASPO), which evaluates individual sentences for more precise preference optimization. By dynamically calculating adaptive rewards at the sentence level based on model predictions, ASPO enhances response content assessment without additional models or parameters. This significantly improves the alignment of multimodal features. Extensive experiments show that ASPO substantially enhances the overall performance of multimodal models.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 34 of its 59 references that are in the map. First posted on arXiv 25 May 2025.

Builds on

34 of 59 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization15 Nov
  2. 2024Improve Vision Language Model Chain-of-thought Reasoning21 Oct
  3. 2024Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement24 May

Other references · 31

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  8. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  9. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  10. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  11. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  12. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
19 more
  1. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 872
  2. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  3. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  4. 2024Self-Rewarding Language Modelscited by 681
  5. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 273
  6. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 264
  7. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 241
  8. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimizationcited by 225
  9. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 198
  10. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 174
  11. 2023Silkie: Preference Distillation for Large Visual Language Modelscited by 130
  12. 2024mDPO: Conditional Preference Optimization for Multimodal Large Language Modelscited by 92
  13. 2024Calibrated Self-Rewarding Vision Language Modelscited by 78
  14. 2023How Many Demonstrations Do You Need for In-context Learning?cited by 52
  15. 2024TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedbackcited by 43
  16. 2023Reformulating Vision-Language Foundation Models and Datasets Towards Universal Multimodal Assistantscited by 25
  17. 2024TPO: Aligning Large Language Models with Multi-branch & Multi-step Preference Treescited by 19
  18. 2024Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignmentcited by 19
  19. 2024TLDR: Token-Level Detective Reward Model for Large Vision Language Modelscited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2026Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelinescited by 1
  2. 2025MedAlign: A Synergistic Framework of Multimodal Preference Optimization and Federated Meta-Cognitive Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts
  2. 2024Direct Preference Optimization with an Offset
  3. 2024Triple Preference Optimization: Achieving Better Alignment using a Single Step Optimization
  4. 2024Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks
  5. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive
  6. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward
  7. 2025MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledge
  8. 2025FocalPO: Enhancing Preference Optimizing by Focusing on Correct Preference Rankings