# RetailKLIP : Finetuning OpenCLIP backbone using metric learning on a single GPU for Zero-shot retail product image classification

arXiv: https://arxiv.org/abs/2312.10282 · HTML: https://mlsanity.com/paper/2312.10282/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 17 references that are in the map.
- First posted on arXiv 16 Dec 2023.

## Builds on (most cited first, 10 of 10)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)
- [ArcFace: Additive Angular Margin Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.07698/) (2018)
- [CLIP Itself is a Strong Fine-tuner: Achieving 85.7% and 88.0% Top-1 Accuracy with ViT-B and ViT-L on ImageNet](https://mlsanity.com/paper/2212.06138/) (2022)
- [RP2K: A Large-Scale Retail Product Dataset for Fine-Grained Image Classification](https://mlsanity.com/paper/2006.12634/) (2020)
- [How to Fine-Tune Vision Models with SGD](https://mlsanity.com/paper/2211.09359/) (2022)
- [Domain invariant hierarchical embedding for grocery products recognition](https://mlsanity.com/paper/1902.00760/) (2019)
- [Bag of Tricks for Retail Product Image Classification](https://mlsanity.com/paper/2001.03992/) (2020)
- [Using Contrastive Learning and Pseudolabels to learn representations for Retail Product Image Classification](https://mlsanity.com/paper/2110.03639/) (2021)

## Built on by (most cited first, 3 of 3)

- [Exploring Fine-grained Retail Product Discrimination with Zero-shot Object Classification Using Vision-Language Models](https://mlsanity.com/paper/2409.14963/) (2024)
- [CLIP-VAD: Exploiting Vision-Language Models for Voice Activity Detection](https://mlsanity.com/paper/2410.14509/) (2024)
- [What Matters for Grocery Product Retrieval with Open Source Vision Language Models](https://mlsanity.com/paper/2605.18029/) (2026)

## Abstract (arXiv)

Retail product or packaged grocery goods images need to classified in various computer vision applications like self checkout stores, supply chain automation and retail execution evaluation. Previous works explore ways to finetune deep models for this purpose. But because of the fact that finetuning a large model or even linear layer for a pretrained backbone requires to run at least a few epochs of gradient descent for every new retail product added in classification range, frequent retrainings are needed in a real world scenario. In this work, we propose finetuning the vision encoder of a CLIP model in a way that its embeddings can be easily used for nearest neighbor based classification, while also getting accuracy close to or exceeding full finetuning. A nearest neighbor based classifier needs no incremental training for new products, thus saving resources and wait time.