# Learning Canonical 3D Object Representation for Fine-Grained Recognition

arXiv: https://arxiv.org/abs/2108.04628 · HTML: https://mlsanity.com/paper/2108.04628/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 80 references that are in the map.
- First posted on arXiv 10 Aug 2021.

## Builds on (most cited first, 30 of 51)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images](https://mlsanity.com/paper/1804.01654/) (2018)
- [3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction](https://mlsanity.com/paper/1604.00449/) (2016)
- [Neural 3D Mesh Renderer](https://mlsanity.com/paper/1711.07566/) (2017)
- [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019)
- [R-FCN: Object Detection via Region-based Fully Convolutional Networks](https://mlsanity.com/paper/1605.06409/) (2016)
- [Learning Category-Specific Mesh Reconstruction from Image Collections](https://mlsanity.com/paper/1803.07549/) (2018)
- [An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution](https://mlsanity.com/paper/1807.03247/) (2018)
- [What Do Single-view 3D Reconstruction Networks Learn?](https://mlsanity.com/paper/1905.03678/) (2019)
- [Bilinear CNNs for Fine-grained Visual Recognition](https://mlsanity.com/paper/1504.07889/) (2015)
- [Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild](https://mlsanity.com/paper/1911.11130/) (2019)
- [Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistency](https://mlsanity.com/paper/1704.06254/) (2017)
- [Self-supervised Single-view 3D Reconstruction via Semantic Consistency](https://mlsanity.com/paper/2003.06473/) (2020)
- [Learning to Navigate for Fine-grained Classification](https://mlsanity.com/paper/1809.00287/) (2018)
- [Pixel2Mesh++: Multi-View 3D Mesh Generation via Deformation](https://mlsanity.com/paper/1908.01491/) (2019)
- [Looking for the Devil in the Details: Learning Trilinear Attention Sampling Network for Fine-grained Image Recognition](https://mlsanity.com/paper/1903.06150/) (2019)
- [Fine-Grained Visual Classification via Progressive Multi-Granularity Training of Jigsaw Patches](https://mlsanity.com/paper/2003.03836/) (2020)
- [Shape and Viewpoint without Keypoints](https://mlsanity.com/paper/2007.10982/) (2020)
- [Unsupervised Learning of Shape and Pose with Differentiable Point Clouds](https://mlsanity.com/paper/1810.09381/) (2018)
- [Polar Transformer Networks](https://mlsanity.com/paper/1709.01889/) (2017)

## Built on by (most cited first, 7 of 7)

- [SIM-Trans: Structure Information Modeling Transformer for Fine-grained Visual Categorization](https://mlsanity.com/paper/2208.14607/) (2022)
- [Fine-Grained Visual Classification using Self Assessment Classifier](https://mlsanity.com/paper/2205.10529/) (2022)
- [Making a Bird AI Expert Work for You and Me](https://mlsanity.com/paper/2112.02747/) (2021)
- [Parsing Objects at a Finer Granularity: A Survey](https://mlsanity.com/paper/2212.13693/) (2022)
- [Not All Birds Look The Same: Identity-Preserving Generation For Birds](https://mlsanity.com/paper/2512.04485/) (2025)
- [Mirror-Fusion Attention for Reflection-Aware Self-Supervised Representation Learning](https://mlsanity.com/paper/2607.00850/) (2026)
- [GeoPose: Patient-agnostic CTA-to-DSA registration through projection-space calibration](https://mlsanity.com/paper/2608.16600/) (2026)

## Abstract (arXiv)

We propose a novel framework for fine-grained object recognition that learns to recover object variation in 3D space from a single image, trained on an image collection without using any ground-truth 3D annotation. We accomplish this by representing an object as a composition of 3D shape and its appearance, while eliminating the effect of camera viewpoint, in a canonical configuration. Unlike conventional methods modeling spatial variation in 2D images only, our method is capable of reconfiguring the appearance feature in a canonical 3D space, thus enabling the subsequent object classifier to be invariant under 3D geometric variation. Our representation also allows us to go beyond existing methods, by incorporating 3D shape variation as an additional cue for object recognition. To learn the model without ground-truth 3D annotation, we deploy a differentiable renderer in an analysis-by-synthesis framework. By incorporating 3D shape and appearance jointly in a deep representation, our method learns the discriminative representation of the object and achieves competitive performance on fine-grained image recognition and vehicle re-identification. We also demonstrate that the performance of 3D shape reconstruction is improved by learning fine-grained shape deformation in a boosting manner.