mlsanityData to 14 Oct 2026

What Matters for Grocery Product Retrieval with Open Source Vision Language Models

arXiv 2605.18029 ↗First posted 18 May 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Multimodal product retrieval (MPR) underpins checkout-free retail and automated inventory systems, yet it demands fine-grained SKU discrimination that standard vision-language benchmarks fail to capture. We present the first systematic zero-shot evaluation of 190 open-source VLMs on the MPR task of the GroceryVision Challenge, isolating pre-training data, architecture, and input resolution. Our analysis yields three actionable findings. (1) Data quality trumps scale. Switching from raw web-scrapes to filtered datasets delivers up to 16.6% accuracy gains, exceeding the benefit of doubling model parameters. (2) Efficient models can win. MobileCLIP-B (150M parameters) outperforms 351M counterparts trained on noisy data. We introduce textitsemantic power density (φ), an efficiency metric that penalizes sub-threshold accuracy. (3) A precision gap persists. State-of-the-art models achieve 94.5% Recall@5 but suffer a 17.5% drop at Recall@1, revealing that contrastive embeddings cluster categories effectively but fail to rank visually similar SKUs. Code and evaluation scripts are available at urlhttps://github.com/upeee/openmpr.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 39 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

33 of 39 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  8. 2022A ConvNet for the 2020scited by 3,579
  9. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  10. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  11. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  12. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
21 more
  1. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
  2. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  3. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  4. 2019Passage Re-ranking with BERTcited by 645
  5. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  6. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  7. 2023Demystifying CLIP Datacited by 270
  8. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 237
  9. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  10. 2021The Efficiency Misnomercited by 98
  11. 2023MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Trainingcited by 96
  12. 2023An Inverse Scaling Law for CLIP Trainingcited by 77
  13. 2025Meta CLIP 2: A Worldwide Scaling Recipecited by 64
  14. 2020Products-10K: A Large-scale Product Recognition Datasetcited by 42
  15. 2024ViTamin: Designing Scalable Vision Models in the Vision-Language Eracited by 39
  16. 2020RP2K: A Large-Scale Retail Product Dataset for Fine-Grained Image Classificationcited by 27
  17. 2023NLLB-CLIP -- train performant multilingual image retrieval model on a budgetcited by 24
  18. 2019Domain invariant hierarchical embedding for grocery products recognitioncited by 5
  19. 2020Bag of Tricks for Retail Product Image Classificationcited by 5
  20. 2023RetailKLIP : Finetuning OpenCLIP backbone using metric learning on a single GPU for Zero-shot retail product image classificationcited by 3
  21. 2025Benchmarking Image Embeddings for E-Commerce: Evaluating Off-the Shelf Foundation Models, Fine-Tuning Strategies and Practical Trade-offscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025An analysis of vision-language models for fabric retrieval
  2. 2025Zero-Shot Retrieval for Scalable Visual Search in a Two-Sided Marketplace
  3. 2022V2L: Leveraging Vision and Vision-language Models into Large-scale Product Retrieval
  4. 2026Ostrakon-VL: Towards Domain-Expert MLLM for Food-Service and Retail Stores
  5. 2026FashionMV: Product-Level Composed Image Retrieval with Multi-View Fashion Data
  6. 2025PRISM: Product Retrieval In Shopping Carts using Hybrid Matching
  7. 2023Food-500 Cap: A Fine-Grained Food Caption Benchmark for Evaluating Vision-Language Models
  8. 2024Exploring Fine-grained Retail Product Discrimination with Zero-shot Object Classification Using Vision-Language Models