Multimodal Hybrid Retrieval-Augmented Generation for Scientific Document Understanding using Open-Source SLMs
On the map: Retrieval-augmented generation → Multimodal RAG
Abstract (arXiv)
Large Language Models tend to hallucinate when answering domain-specific ques tions from scientific documents without prior fine-tuning. Currently, methods such as Retrieval-Augmented Generation partially solve this problem but face different challenges: limited context knowledge, difference between sparse and dense retrieval, and retrieval noise. This paper presents an Advanced Multimodal Retrieval-Augmented Generation system that aims to solve those challenges and im prove the accuracy of information extraction. The proposed architecture introduces a multimodal ingestion pipeline that leverages an open-source Vision-Language Model (Qwen2-VL-2B-Instruct) to generate textual summaries of tables and fig ures. The retrieval phase integrates HNSW-based semantic search with GIN-based lexical search, unified through Reciprocal Rank Fusion and refined using Cross Encoder reranking to minimize retrieval noise. To ensure conversational coherence across multi-turn interactions, a Query Condenser module is employed. Evaluation is conducted by independently assessing the ingestion, retrieval and generation stages using the MMLongBench benchmark, a BeIR-format synthetic dataset and the DeepEval framework. Moreover, results demonstrate a 157% improvement in retrieval quality over a Naive-RAG baseline, with only 50 ms additional la tency, while Qwen2-VL-2B-Instruct achieved results comparable to cloud-based models in BERTScore. These findings validate that open-source optimized SLMs, paired with advanced retrieval strategies, can provide competitive performance for document understanding without relying on cloud-based models.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 11 of its 14 references that are in the map. First posted on arXiv 6 Jul 2026.
Builds on
11 of 14 references.
References · 11
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
- 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
- 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 861
- 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applicationscited by 431
- 2025EmbeddingGemma: Powerful and Lightweight Text Representationscited by 139
- 2018ROUGE 2.0: Updated and Improved Measures for Evaluation of Summarization Taskscited by 67
- 2024Efficient Inverted Indexes for Approximate Retrieval over Learned Sparse Representationscited by 24
- 2024Optimizing Query Generation for Enhanced Document Retrieval in RAGcited by 10
- 2025Assessing RAG and HyDE on 1B vs. 4B-Parameter Gemma LLMs for Personal Assistants Integretioncited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text
- 2025Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understanding
- 2025HiPerRAG: High-Performance Retrieval Augmented Generation for Scientific Insights
- 2025OMGM: Orchestrate Multiple Granularities and Modalities for Efficient Multimodal Retrieval
- 2025RAG-Anything: All-in-One RAG Framework
- 2024Retrieval Meets Reasoning: Even High-school Textbook Knowledge Benefits Multimodal Reasoning
- 2025HKRAG: Holistic Knowledge Retrieval-Augmented Generation Over Visually-Rich Documents
- 2025Towards Mixed-Modal Retrieval for Universal Retrieval-Augmented Generation