mlsanityData to 14 Oct 2026

CUE-M: Contextual Understanding and Enhanced Search with Multimodal Large Language Model

arXiv 2411.12287 ↗First posted 19 Nov 2024cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

also in 1 more
Abstract (arXiv)

The integration of Retrieval-Augmented Generation (RAG) with Multimodal Large Language Models (MLLMs) has revolutionized information retrieval and expanded the practical applications of AI. However, current systems struggle in accurately interpreting user intent, employing diverse retrieval strategies, and effectively filtering unintended or inappropriate responses, limiting their effectiveness. This paper introduces Contextual Understanding and Enhanced Search with MLLM (CUE-M), a novel multimodal search framework that addresses these challenges through a multi-stage pipeline comprising image context enrichment, intent refinement, contextual query generation, external API integration, and relevance-based filtering. CUE-M incorporates a robust filtering pipeline combining image-based, text-based, and multimodal classifiers, dynamically adapting to instance- and category-specific concern defined by organizational policies. Extensive experiments on real-word datasets and public benchmarks on knowledge-based VQA and safety demonstrated that CUE-M outperforms baselines and establishes new state-of-the-art results, advancing the capabilities of multimodal retrieval systems.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 55 references that are in the map. First posted on arXiv 19 Nov 2024.

Builds on

38 of 55 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024EchoSight: Advancing Visual-Language Models with Wiki Knowledge17 Jul
  2. 2024Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs23 Apr

Other references · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  8. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  9. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  10. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  11. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  12. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
24 more
  1. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 301
  2. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  3. 2023PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimizationcited by 233
  4. 2020X-Linear Attention Networks for Image Captioningcited by 198
  5. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 172
  6. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 166
  7. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 131
  8. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 126
  9. 2021Entity-level Factual Consistency of Abstractive Text Summarizationcited by 118
  10. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performancecited by 117
  11. 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categoriescited by 107
  12. 2014Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)cited by 94
  13. 2024Safety of Multimodal Large Language Models on Images and Textscited by 75
  14. 2024GOAT-Bench: Safety Insights to Large Multimodal Models through Meme-Based Social Abusecited by 52
  15. 2024Unveiling the Safety of GPT-4o: An Empirical Study using Jailbreak Attackscited by 38
  16. 2022Contrastive Language-Image Pre-Training with Knowledge Graphscited by 34
  17. 2024CoCA: Regaining Safety-awareness of Multimodal Large Language Models with Constitutional Calibrationcited by 31
  18. 2023EDIS: Entity-Driven Image Search over Multimodal Web Contentcited by 27
  19. 2021Passage Retrieval for Outside-Knowledge Visual Question Answeringcited by 25
  20. 2023End-to-end Knowledge Retrieval with Multi-modal Queriescited by 23
  21. 2021Telling the What while Pointing to the Where: Multimodal Queries for Image Retrievalcited by 12
  22. 2021He Thinks He Knows Better than the Doctors: BERT for Event Factuality Fails on Pragmaticscited by 12
  23. 2024Grounding Language Models for Visual Entity Recognitioncited by 12
  24. 2024Captions Are Worth a Thousand Words: Enhancing Product Retrieval with Pretrained Image-to-Text Modelscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Benchmarking Retrieval-Augmented Generation in Multi-Modal Contexts
  2. 2024Leveraging Large Language Models for Multimodal Search
  3. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation
  4. 2026RETLLM: Training and Data-Free MLLMs for Multimodal Information Retrieval
  5. 2025Re-ranking the Context for Multimodal Retrieval Augmented Generation
  6. 2024MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs
  7. 2024Evaluating Cost-Accuracy Trade-offs in Multimodal Search Relevance Judgements
  8. 2024MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Training