# Reliability-Prioritized Fine-Grained Generation in Multimodal Large

arXiv: https://arxiv.org/abs/2606.29573 · HTML: https://mlsanity.com/paper/2606.29573/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 27 of its 47 references that are in the map.
- First posted on arXiv 28 Jun 2026.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 27 of 27)

- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://mlsanity.com/paper/2311.12793/) (2023)
- [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://mlsanity.com/paper/2405.14734/) (2024)
- [Aligning Large Multimodal Models with Factually Augmented RLHF](https://mlsanity.com/paper/2309.14525/) (2023)
- [Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding](https://mlsanity.com/paper/2311.16922/) (2023)
- [Ferret: Refer and Ground Anything Anywhere at Any Granularity](https://mlsanity.com/paper/2310.07704/) (2023)
- [GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning](https://mlsanity.com/paper/2507.01006/) (2025)
- [GLaMM: Pixel Grounding Large Multimodal Model](https://mlsanity.com/paper/2311.03356/) (2023)
- [RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback](https://mlsanity.com/paper/2312.00849/) (2023)
- [MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?](https://mlsanity.com/paper/2408.13257/) (2024)
- [Benchmarking and Improving Detail Image Caption](https://mlsanity.com/paper/2405.19092/) (2024)
- [CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era](https://mlsanity.com/paper/2503.12329/) (2025)
- [CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness](https://mlsanity.com/paper/2502.14914/) (2025)
- [Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning](https://mlsanity.com/paper/2503.07906/) (2025)
- [Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation](https://mlsanity.com/paper/2504.14988/) (2025)
- [Robust Preference Optimization via Dynamic Target Margins](https://mlsanity.com/paper/2506.03690/) (2025)
- [ProTeCt: Prompt Tuning for Taxonomic Open Set Classification](https://mlsanity.com/paper/2306.02240/) (2023)
- [Fine-R1: Make Multi-modal LLMs Excel in Fine-Grained Visual Recognition by Chain-of-Thought Reasoning](https://mlsanity.com/paper/2602.07605/) (2026)
- [Mitigating Hallucination in Multimodal LLMs with Layer Contrastive Decoding](https://mlsanity.com/paper/2509.25177/) (2025)
- [FINER: MLLMs Hallucinate under Fine-grained Negative Queries](https://mlsanity.com/paper/2603.17662/) (2026)
- [FREAK: A Fine-grained Hallucination Evaluation Benchmark for Advanced MLLMs](https://mlsanity.com/paper/2603.19765/) (2026)
- [Modality Alignment across Trees on Heterogeneous Hyperbolic Manifolds](https://mlsanity.com/paper/2510.27391/) (2025)

## Abstract (arXiv)

Multimodal large language models (MLLMs) are increasingly expected to generate fine-grained descriptions of visual content. However, we observe and theoretically show that generating fine-grained responses poses a reliability challenge, textiti.e., fine-grained generation is more error-prone than coarse-grained generation. This phenomenon suggests that models should generate the finest description that remains reliable rather than simply produce more specific outputs. To investigate this problem, we develop textscGranFact, a granularity-aware benchmark consisting of expert-verified multi-object images with coarse-to-fine category annotations. Then, we design a hierarchy-aware evaluation algorithm, which assesses both whether model predictions are visually correct and how specific the correct predictions are. We also propose a reliability-prioritized preference optimization method based on Direct Preference Optimization, which penalizes unreliable fine-grained claims while rewarding reliable specificity. Experiments on textscGranFact show that our method improves fine-grained generation while preserving reliability. Code and data are available hrefhttps://github.com/WeiWu2025/GranFacthere.