# MultiGrain: a unified image embedding for classes and instances

arXiv: https://arxiv.org/abs/1902.05509 · HTML: https://mlsanity.com/paper/1902.05509/

## Key facts

- Cited by 85 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Feb 2019.

## Built on by (most cited first, 30 of 85)

- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference](https://mlsanity.com/paper/2104.01136/) (2021)
- [Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models](https://mlsanity.com/paper/2212.03860/) (2022)
- [Early Convolutions Help Transformers See Better](https://mlsanity.com/paper/2106.14881/) (2021)
- [ResNet strikes back: An improved training procedure in timm](https://mlsanity.com/paper/2110.00476/) (2021)
- [XCiT: Cross-Covariance Image Transformers](https://mlsanity.com/paper/2106.09681/) (2021)
- [Focal Self-attention for Local-Global Interactions in Vision Transformers](https://mlsanity.com/paper/2107.00641/) (2021)
- [RTMDet: An Empirical Study of Designing Real-Time Object Detectors](https://mlsanity.com/paper/2212.07784/) (2022)
- [A Self-Supervised Descriptor for Image Copy Detection](https://mlsanity.com/paper/2202.10261/) (2022)
- [DaViT: Dual Attention Vision Transformers](https://mlsanity.com/paper/2204.03645/) (2022)
- [Visformer: The Vision-friendly Transformer](https://mlsanity.com/paper/2104.12533/) (2021)
- [MetaFormer Baselines for Vision](https://mlsanity.com/paper/2210.13452/) (2022)
- [Semi-Supervised Learning of Visual Features by Non-Parametrically Predicting View Assignments with Support Samples](https://mlsanity.com/paper/2104.13963/) (2021)
- [OmniMAE: Single Model Masked Pretraining on Images and Videos](https://mlsanity.com/paper/2206.08356/) (2022)
- [Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervision](https://mlsanity.com/paper/2202.08360/) (2022)
- [Masked Frequency Modeling for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2206.07706/) (2022)
- [Corrupted Image Modeling for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2202.03382/) (2022)
- [ProtoPFormer: Concentrating on Prototypical Parts in Vision Transformers for Interpretable Image Recognition](https://mlsanity.com/paper/2208.10431/) (2022)
- [Which Tokens to Use? Investigating Token Reduction in Vision Transformers](https://mlsanity.com/paper/2308.04657/) (2023)
- [Use All The Labels: A Hierarchical Multi-Label Contrastive Learning Framework](https://mlsanity.com/paper/2204.13207/) (2022)
- [CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings](https://mlsanity.com/paper/2106.03143/) (2021)
- [Correlation Verification for Image Retrieval](https://mlsanity.com/paper/2204.01458/) (2022)
- [GeneCIS: A Benchmark for General Conditional Image Similarity](https://mlsanity.com/paper/2306.07969/) (2023)
- [The 2021 Image Similarity Dataset and Challenge](https://mlsanity.com/paper/2106.09672/) (2021)
- [Data augmentation in Bayesian neural networks and the cold posterior effect](https://mlsanity.com/paper/2106.05586/) (2021)
- [Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Error](https://mlsanity.com/paper/2105.13343/) (2021)
- [A Battle of Network Structures: An Empirical Study of CNN, Transformer, and MLP](https://mlsanity.com/paper/2108.13002/) (2021)

## Abstract (arXiv)

MultiGrain is a network architecture producing compact vector representations that are suited both for image classification and particular object retrieval. It builds on a standard classification trunk. The top of the network produces an embedding containing coarse and fine-grained information, so that images can be recognized based on the object class, particular object, or if they are distorted copies. Our joint training is simple: we minimize a cross-entropy loss for classification and a ranking loss that determines if two images are identical up to data augmentation, with no need for additional labels. A key component of MultiGrain is a pooling layer that takes advantage of high-resolution images with a network trained at a lower resolution. When fed to a linear classifier, the learned embeddings provide state-of-the-art classification accuracy. For instance, we obtain 79.4% top-1 accuracy with a ResNet-50 learned on Imagenet, which is a +1.8% absolute improvement over the AutoAugment method. When compared with the cosine similarity, the same embeddings perform on par with the state-of-the-art for image retrieval at moderate resolutions.