mlsanityData to 14 Oct 2026

POPEN: Preference-Based Optimization and Ensemble for LVLM-Based Reasoning Segmentation

arXiv 2504.00640 ↗First posted 1 Apr 2025cs.CVcited by 22

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Existing LVLM-based reasoning segmentation methods often suffer from imprecise segmentation results and hallucinations in their text responses. This paper introduces POPEN, a novel framework designed to address these issues and achieve improved results. POPEN includes a preference-based optimization method to finetune the LVLM, aligning it more closely with human preferences and thereby generating better text responses and segmentation results. Additionally, POPEN introduces a preference-based ensemble method for inference, which integrates multiple outputs from the LVLM using a preference-score-based attention mechanism for refinement. To better adapt to the segmentation task, we incorporate several task-specific designs in our POPEN framework, including a new approach for collecting segmentation preference data with a curriculum learning mechanism, and a novel preference optimization loss to refine the segmentation capability of the LVLM. Experiments demonstrate that our method achieves state-of-the-art performance in reasoning segmentation, exhibiting minimal hallucination in text responses and the highest segmentation accuracy compared to previous advanced methods like LISA and PixelLM. Project page is https://lanyunzhu.site/POPEN/

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 55 of its 63 references that are in the map. First posted on arXiv 1 Apr 2025.

Builds on

55 of 63 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024VISA: Reasoning Video Object Segmentation via Large Language Models16 Jul
  2. 2023PixelLM: Pixel Reasoning with Large Multimodal Model4 Dec
  3. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug

Other references · 52

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  2. 2023Segment Anythingcited by 8,234
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  7. 2017Deep reinforcement learning from human preferencescited by 3,912
  8. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  10. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  11. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,161
  12. 2018On the Spectral Bias of Neural Networkscited by 1,341
38 more
  1. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  2. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  3. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 790
  4. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  5. 2018Object Hallucination in Image Captioningcited by 669
  6. 2014Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFscited by 554
  7. 2023Segment Everything Everywhere All at Oncecited by 548
  8. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  9. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 478
  10. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
  11. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 412
  12. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  13. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 348
  14. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 323
  15. 2022Generalized Decoding for Pixel, Image, and Languagecited by 294
  16. 2023Preference Ranking Optimization for Human Alignmentcited by 285
  17. 2023GRES: Generalized Referring Expression Segmentationcited by 254
  18. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 240
  19. 2020Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentationcited by 204
  20. 2018A mathematical theory of semantic development in deep neural networkscited by 186
  21. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 174
  22. 2023GSVA: Generalized Segmentation via Multimodal Large Language Modelscited by 154
  23. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Rewardcited by 147
  24. 2023PACO: Parts and Attributes of Common Objectscited by 146
  25. 2024Direct Preference Optimization with an Offsetcited by 122
  26. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decodingcited by 118
  27. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMscited by 114
  28. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Modelcited by 101
  29. 2024GROUNDHOG: Grounding Large Language Models to Holistic Segmentationcited by 70
  30. 2024SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentationcited by 69
  31. 2021Learning Statistical Texture for Semantic Segmentationcited by 51
  32. 2023LLaFS: When Large Language Models Meet Few-Shot Segmentationcited by 48
  33. 2023Continual Semantic Segmentation with Automatic Memory Sample Selectioncited by 39
  34. 2023Structural and Statistical Texture Knowledge Distillation for Semantic Segmentationcited by 36
  35. 2023Ultra-High Resolution Segmentation with Ultra-Rich Context: A Novel Benchmarkcited by 22
  36. 2023FVP: Fourier Visual Prompting for Source-Free Unsupervised Domain Adaptation of Medical Image Segmentationcited by 21
  37. 2024Hybrid Mamba for Few-Shot Segmentationcited by 21
  38. 2024Discrete Latent Perspective Learning for Segmentation and Detectioncited by 16

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 17.

22 papers.

Sort:

By area:

  1. 2025PixelThink: Towards Efficient Chain-of-Pixel Reasoningcited by 28
  2. 2025Multimodal Referring Segmentation: A Surveycited by 22
  3. 2025UGround: Towards Unified Visual Grounding with Unrolled Transformerscited by 14
  4. 2025Text4Seg++: Advancing Image Segmentation via Generative Language Modelingcited by 9
  5. 2026LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generationcited by 6
  6. 2026StreamCacheVGGT: Streaming Visual Geometry Transformers with Robust Scoring and Hybrid Cache Compressioncited by 5
  7. 2025RAVEN++: Pinpointing Fine-Grained Violations in Advertisement Videos with Active Reinforcement Reasoningcited by 4
  8. 2026DR2Seg: Decomposed Two-Stage Rollouts for Efficient Reasoning Segmentation in Multimodal Large Language Modelscited by 3
  9. 2026HD-VGGT: High-Resolution Visual Geometry Transformercited by 3
  10. 2026StAR: Segment Anything Reasonercited by 2
  11. 2026Robust 4D Visual Geometry Transformer with Uncertainty-Aware Priorscited by 2
  12. 2026B-GRTO: Bootstrapped Group Relative Tool Optimization for Referring Segmentationcited by 1
10 more
  1. 2026Prefer-DAS: Learning from Local Preferences and Sparse Prompts for Domain Adaptive Segmentation of Electron Microscopycited by 0
  2. 2026Visual Attention Drifts,but Anchors Hold:Mitigating Hallucination in Multimodal Large Language Models via Cross-Layer Visual Anchorscited by 0
  3. 2026Learning to Label: A Reinforced Self-Evolving Framework for Semi-supervised Referring Expression Segmentationcited by 0
  4. 2026From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Modelscited by 0
  5. 2026GEAR-Seg: A Grounded Explainable Agent for Reasoning Segmentation and Data Enginecited by 0
  6. 2026Better, Stronger, Faster, and Broader: Structured All-Mask Prediction for MLLM-Based Segmentationcited by 0
  7. 2026DRAgent: Discriminative Reasoning Agent for Referring Expression Segmentationcited by 0
  8. 2026Position Is All You Need: A Free Lunch Token Compression Strategy for MLLM-based Referring Expression Segmentationcited by 0
  9. 20260.5%>100%: Bidirectional Reciprocal Learning for Referring Image Segmentationcited by 0
  10. 2026Skip the Talk, Re-Focus on Vision: Latent Reasoning for Reasoning Segmentation in Multimodal Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Knowing the Answer Isn't Enough: Fixing Reasoning Path Failures in LVLMs
  2. 2026Reason Twice: Segmentation via Candidate Discovery and Comparative Reasoning
  3. 2026HiPO: Hierarchical Preference Optimization for Adaptive Reasoning in LLMs
  4. 2025LVLM_CSP: Accelerating Large Vision Language Models via Clustering, Scattering, and Pruning for Reasoning Segmentation
  5. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization
  6. 2025CoPRS: Learning Positional Prior from Chain-of-Thought for Reasoning Segmentation
  7. 2026CoT-Seg: Rethinking Segmentation with Chain-of-Thought Reasoning and Self-Correction
  8. 2026Reasoning Matters: Mitigate Hallucination in Multimodal Large Reasoning Models via Reasoning-Conditioned Preference Optimization