mlsanityData to 14 Oct 2026

ROSE: Retrieval-Oriented Segmentation Enhancement

arXiv 2604.14147 ↗First posted 15 Apr 2026cs.CVcited by 3

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Existing segmentation models based on multimodal large language models (MLLMs), such as LISA, often struggle with novel or emerging entities due to their inability to incorporate up-to-date knowledge. To address this challenge, we introduce the Novel Emerging Segmentation Task (NEST), which focuses on segmenting (i) novel entities that MLLMs fail to recognize due to their absence from training data, and (ii) emerging entities that exist within the model's knowledge but demand up-to-date external information for accurate recognition. To support the study of NEST, we construct a NEST benchmark using an automated pipeline that generates news-related data samples for comprehensive evaluation. Additionally, we propose ROSE: Retrieval-Oriented Segmentation Enhancement, a plug-and-play framework designed to augment any MLLM-based segmentation model. ROSE comprises four key components. First, an Internet Retrieval-Augmented Generation module is introduced to employ user-provided multimodal inputs to retrieve real-time web information. Then, a Textual Prompt Enhancer enriches the model with up-to-date information and rich background knowledge, improving the model's perception ability for emerging entities. Furthermore, a Visual Prompt Enhancer is proposed to compensate for MLLMs' lack of exposure to novel entities by leveraging internet-sourced images. To maintain efficiency, a WebSense module is introduced to intelligently decide when to invoke retrieval mechanisms based on user input. Experimental results demonstrate that ROSE significantly boosts performance on the NEST benchmark, outperforming a strong Gemini-2.0 Flash-based retrieval baseline by 19.2 in gIoU.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 36 references that are in the map. First posted on arXiv 15 Apr 2026.

Builds on

31 of 36 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024SearchLVLMs: A Plug-and-Play Framework for Augmenting Large Vision-Language Models by Searching Up-to-Date Internet Knowledge23 May
  2. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory10 Dec
  3. 2022Retrieval-Augmented Multimodal Language Modeling22 Nov

Other references · 28

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Segment Anythingcited by 8,234
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2024GPT-4o System Cardcited by 4,588
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  8. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  9. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  10. 2016Modeling Context in Referring Expressionscited by 1,063
  11. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 929
  12. 2023Large Language Modelscited by 780
16 more
  1. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  2. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  3. 2023Segment Everything Everywhere All at Oncecited by 548
  4. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 452
  5. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  6. 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 298
  7. 2016Segmentation from Natural Language Expressionscited by 257
  8. 2023GRES: Generalized Referring Expression Segmentationcited by 254
  9. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 240
  10. 2022Internet-augmented language models through few-shot prompting for open-domain question answeringcited by 131
  11. 2017Recurrent Multimodal Interaction for Referring Image Segmentationcited by 125
  12. 2022VLT: Vision-Language Transformer and Query Generation for Referring Segmentationcited by 111
  13. 2023See, Say, and Segment: Teaching LMMs to Overcome False Premisescited by 41
  14. 2025MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentationcited by 35
  15. 2025Multimodal Referring Segmentation: A Surveycited by 22
  16. 2026GREx: Generalized Referring Expression Segmentation, Comprehension, and Generationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026From Web to Pixels: Bringing Agentic Search into Visual Perceptioncited by 3
  2. 2026Seek to Segment: Active Perception for Panoramic Referring Segmentationcited by 0
  3. 2026EviRover: Reinforcing Agentic Perception Beyond a Glancecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing Retrieval Augmented Generation with Hierarchical Text Segmentation Chunking
  2. 2024Empowering Segmentation Ability to Multi-modal Large Language Models
  3. 2025SAGE: A Framework of Precise Retrieval for RAG
  4. 2024MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Training
  5. 2026Retrieve and Segment: Are a Few Examples Enough to Bridge the Supervision Gap in Open-Vocabulary Segmentation?
  6. 2026Retrieval or Representation? Reassessing Benchmark Gaps in Multilingual and Visually Rich RAG
  7. 2025Enhancing Retrieval-Augmented Generation with Two-Stage Retrieval: FlashRank Reranking and Query Expansion
  8. 2025VaccineRAG: Boosting Multimodal Large Language Models' Immunity to Harmful RAG Samples