# A Visual RAG Pipeline for Few-Shot Fine-Grained Product Classification

arXiv: https://arxiv.org/abs/2504.11838 · HTML: https://mlsanity.com/paper/2504.11838/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 16 of its 29 references that are in the map.
- First posted on arXiv 16 Apr 2025.

## Placed in

- Retrieval-augmented generation / [Multimodal RAG](https://mlsanity.com/rag/multimodal-rag/)

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents](https://mlsanity.com/paper/2410.10594/) (2024)
- [Large-scale Multi-Modal Pre-trained Models: A Comprehensive Survey](https://mlsanity.com/paper/2302.10035/) (2023)
- [Fine-Grained Image Analysis with Deep Learning: A Survey](https://mlsanity.com/paper/2111.06119/) (2021)
- [MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Training](https://mlsanity.com/paper/2407.21439/) (2024)
- [Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation](https://mlsanity.com/paper/2502.08826/) (2025)
- [FETA: Towards Specializing Foundation Models for Expert Task Applications](https://mlsanity.com/paper/2209.03648/) (2022)
- [A Hierarchical Grocery Store Image Dataset with Visual and Semantic Labels](https://mlsanity.com/paper/1901.00711/) (2019)
- [Can Visual Language Models Replace OCR-Based Visual Question Answering Pipelines in Production? A Case Study in Retail](https://mlsanity.com/paper/2408.15626/) (2024)
- [Retail-786k: a Large-Scale Dataset for Visual Entity Matching](https://mlsanity.com/paper/2309.17164/) (2023)
- [Creation and Evaluation of a Food Product Image Dataset for Product Property Extraction](https://mlsanity.com/paper/2411.10591/) (2024)

## Built on by (most cited first, 1 of 1)

- [ReCollab: Retrieval-Augmented LLMs for Cooperative Ad-hoc Teammate Modeling](https://mlsanity.com/paper/2512.22129/) (2025)

## Abstract (arXiv)

Despite the rapid evolution of learning and computer vision algorithms, Fine-Grained Classification (FGC) still poses an open problem in many practically relevant applications. In the retail domain, for example, the identification of fast changing and visually highly similar products and their properties are key to automated price-monitoring and product recommendation. This paper presents a novel Visual RAG pipeline that combines the Retrieval Augmented Generation (RAG) approach and Vision Language Models (VLMs) for few-shot FGC. This Visual RAG pipeline extracts product and promotion data in advertisement leaflets from various retailers and simultaneously predicts fine-grained product ids along with price and discount information. Compared to previous approaches, the key characteristic of the Visual RAG pipeline is that it allows the prediction of novel products without re-training, simply by adding a few class samples to the RAG database. Comparing several VLM back-ends like GPT-4o [23], GPT-4o-mini [24], and Gemini 2.0 Flash [10], our approach achieves 86.8% accuracy on a diverse dataset.