# Towards Fine-grained Visual Representations by Combining Contrastive Learning with Image Reconstruction and Attention-weighted Pooling

arXiv: https://arxiv.org/abs/2104.04323 · HTML: https://mlsanity.com/paper/2104.04323/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 39 references that are in the map.
- First posted on arXiv 9 Apr 2021.

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)
- [Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?](https://mlsanity.com/paper/2003.11539/) (2020)
- [Rethinking ImageNet Pre-training](https://mlsanity.com/paper/1811.08883/) (2018)
- [Audio-Visual Scene Analysis with Self-Supervised Multisensory Features](https://mlsanity.com/paper/1804.03641/) (2018)
- [Models Genesis: Generic Autodidactic Models for 3D Medical Image Analysis](https://mlsanity.com/paper/1908.06912/) (2019)
- [3D Self-Supervised Methods for Medical Imaging](https://mlsanity.com/paper/2006.03829/) (2020)
- [Learning and Evaluating Representations for Deep One-class Classification](https://mlsanity.com/paper/2011.02578/) (2020)
- [Cross-Domain Adaptation for Animal Pose Estimation](https://mlsanity.com/paper/1908.05806/) (2019)
- [The Devil is in the Channels: Mutual-Channel Loss for Fine-Grained Image Classification](https://mlsanity.com/paper/2002.04264/) (2020)
- [Multimodal Self-Supervised Learning for Medical Image Analysis](https://mlsanity.com/paper/1912.05396/) (2019)
- [CheXtransfer: Performance and Parameter Efficiency of ImageNet Models for Chest X-Ray Interpretation](https://mlsanity.com/paper/2101.06871/) (2021)
- [Self-Supervision Closes the Gap Between Weak and Strong Supervision in Histology](https://mlsanity.com/paper/2012.03583/) (2020)
- [Improving Landmark Localization with Semi-Supervised Learning](https://mlsanity.com/paper/1709.01591/) (2017)
- [Attribute-Aware Attention Model for Fine-grained Representation Learning](https://mlsanity.com/paper/1901.00392/) (2019)
- [Deep Learning for Fine-Grained Image Analysis: A Survey](https://mlsanity.com/paper/1907.03069/) (2019)
- [Self-supervised audio representation learning for mobile devices](https://mlsanity.com/paper/1905.11796/) (2019)
- [Fine-Grain Few-Shot Vision via Domain Knowledge as Hyperspherical Priors](https://mlsanity.com/paper/2005.11450/) (2020)

## Built on by (most cited first, 13 of 13)

- [How Transferable Are Self-supervised Features in Medical Image Classification Tasks?](https://mlsanity.com/paper/2108.10048/) (2021)
- [The Clever Hans Effect in Unsupervised Learning](https://mlsanity.com/paper/2408.08041/) (2024)
- [Label-Only Model Inversion Attacks via Knowledge Transfer](https://mlsanity.com/paper/2310.19342/) (2023)
- [Model Inversion Robustness: Can Transfer Learning Help?](https://mlsanity.com/paper/2405.05588/) (2024)
- [On the Vulnerability of Skip Connections to Model Inversion Attacks](https://mlsanity.com/paper/2409.01696/) (2024)
- [Keep Your Friends Close & Enemies Farther: Debiasing Contrastive Learning with Spatial Priors in 3D Radiology Images](https://mlsanity.com/paper/2211.08643/) (2022)
- [Transfer Learning for Segmentation Problems: Choose the Right Encoder and Skip the Decoder](https://mlsanity.com/paper/2207.14508/) (2022)
- [Robust Alzheimer's Progression Modeling using Cross-Domain Self-Supervised Deep Learning](https://mlsanity.com/paper/2211.08559/) (2022)
- [Exploiting Supervised Poison Vulnerability to Strengthen Self-Supervised Defense](https://mlsanity.com/paper/2409.08509/) (2024)
- [Revisiting Model Inversion Evaluation: From Misleading Standards to Reliable Privacy Assessment](https://mlsanity.com/paper/2505.03519/) (2025)
- [Contrastive Learning Boosts Deterministic and Generative Models for Weather Data](https://mlsanity.com/paper/2603.24744/) (2026)
- [AdaViT: Adaptive Vision Transformer for Flexible Pretrain and Finetune with Variable 3D Medical Image Modalities](https://mlsanity.com/paper/2504.03589/) (2025)
- [Template Collapse and Information-Theoretic Limits in Camera rPPG Pulse Morphology Restoration](https://mlsanity.com/paper/2606.03802/) (2026)

## Abstract (arXiv)

This paper presents Contrastive Reconstruction, ConRec - a self-supervised learning algorithm that obtains image representations by jointly optimizing a contrastive and a self-reconstruction loss. We showcase that state-of-the-art contrastive learning methods (e.g. SimCLR) have shortcomings to capture fine-grained visual features in their representations. ConRec extends the SimCLR framework by adding (1) a self-reconstruction task and (2) an attention mechanism within the contrastive learning task. This is accomplished by applying a simple encoder-decoder architecture with two heads. We show that both extensions contribute towards an improved vector representation for images with fine-grained visual features. Combining those concepts, ConRec outperforms SimCLR and SimCLR with Attention-Pooling on fine-grained classification datasets.