# Towards Viewpoint-Invariant Visual Recognition via Adversarial Training

arXiv: https://arxiv.org/abs/2307.10235 · HTML: https://mlsanity.com/paper/2307.10235/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 69 references that are in the map.
- First posted on arXiv 16 Jul 2023.

## Builds on (most cited first, 30 of 49)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Theoretically Principled Trade-off between Robustness and Accuracy](https://mlsanity.com/paper/1901.08573/) (2019)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://mlsanity.com/paper/1912.02781/) (2019)
- [Ensemble Adversarial Training: Attacks and Defenses](https://mlsanity.com/paper/1705.07204/) (2017)
- [Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction](https://mlsanity.com/paper/2109.00512/) (2021)
- [Adversarial Training for Free!](https://mlsanity.com/paper/1904.12843/) (2019)
- [Making Convolutional Networks Shift-Invariant Again](https://mlsanity.com/paper/1904.11486/) (2019)
- [ABO: Dataset and Benchmarks for Real-World 3D Object Understanding](https://mlsanity.com/paper/2110.06199/) (2021)
- [Do Adversarially Robust ImageNet Models Transfer Better?](https://mlsanity.com/paper/2007.08489/) (2020)
- [Adversarial Training Methods for Semi-Supervised Text Classification](https://mlsanity.com/paper/1605.07725/) (2016)
- [Recent Advances in Adversarial Training for Adversarial Robustness](https://mlsanity.com/paper/2102.01356/) (2021)

## Built on by (most cited first, 19 of 19)

- [Towards Transferable Targeted 3D Adversarial Attack in the Physical World](https://mlsanity.com/paper/2312.09558/) (2023)
- [Omniview-Tuning: Boosting Viewpoint Invariance of Vision-Language Pre-training Models](https://mlsanity.com/paper/2404.12139/) (2024)
- [UV-Attack: Physical-World Adversarial Attacks for Person Detection via Dynamic-NeRF-based UV Mapping](https://mlsanity.com/paper/2501.05783/) (2025)
- [Improving Viewpoint Robustness for Visual Recognition via Adversarial Training](https://mlsanity.com/paper/2307.11528/) (2023)
- [Adversarial Training: A Survey](https://mlsanity.com/paper/2410.15042/) (2024)
- [CHARM3R: Towards Unseen Camera Height Robust Monocular 3D Detector](https://mlsanity.com/paper/2508.11185/) (2025)
- [AdvDreamer Unveils: Are Vision-Language Models Truly Ready for Real-World 3D Variations?](https://mlsanity.com/paper/2412.03002/) (2024)
- [Not all Views are Created Equal: Analyzing Viewpoint Instabilities in Vision Foundation Models](https://mlsanity.com/paper/2412.19920/) (2024)
- [Test-Time Canonicalization by Foundation Models for Robust Perception](https://mlsanity.com/paper/2507.10375/) (2025)
- [MVBoost: Boost 3D Reconstruction with Multi-View Refinement](https://mlsanity.com/paper/2411.17772/) (2024)
- [View Transformation Robustness for Multi-View 3D Object Reconstruction with Reconstruction Error-Guided View Selection](https://mlsanity.com/paper/2412.11428/) (2024)
- [Evaluating Foundation Models' 3D Understanding Through Multi-View Correspondence Analysis](https://mlsanity.com/paper/2512.11574/) (2025)
- [Spectral Defense Against Resource-Targeting Attack in 3D Gaussian Splatting](https://mlsanity.com/paper/2603.12796/) (2026)
- [MoViD: View-Invariant 3D Human Pose Estimation via Motion-View Disentanglement](https://mlsanity.com/paper/2604.03299/) (2026)
- [Toddlers' Active Gaze Behavior Supports Self-Supervised Object Learning](https://mlsanity.com/paper/2411.01969/) (2024)
- [Active View Selection for Scene-level Multi-view Crowd Counting and Localization with Limited Labeling Budget](https://mlsanity.com/paper/2509.16684/) (2025)
- [Challenging Vision-Language Models with Physically Deployable Multimodal Semantic Lighting Attacks](https://mlsanity.com/paper/2604.12833/) (2026)
- [CrossVL: Complexity-Aware Feature Routing and Paired Curriculum for Cross-View Vision-Language Detection](https://mlsanity.com/paper/2605.09802/) (2026)
- [CAD-Free Learning of Spacecraft Pose Estimators via NeRF-Based Augmentations](https://mlsanity.com/paper/2605.19649/) (2026)

## Abstract (arXiv)

Visual recognition models are not invariant to viewpoint changes in the 3D world, as different viewing directions can dramatically affect the predictions given the same object. Although many efforts have been devoted to making neural networks invariant to 2D image translations and rotations, viewpoint invariance is rarely investigated. As most models process images in the perspective view, it is challenging to impose invariance to 3D viewpoint changes based only on 2D inputs. Motivated by the success of adversarial training in promoting model robustness, we propose Viewpoint-Invariant Adversarial Training (VIAT) to improve viewpoint robustness of common image classifiers. By regarding viewpoint transformation as an attack, VIAT is formulated as a minimax optimization problem, where the inner maximization characterizes diverse adversarial viewpoints by learning a Gaussian mixture distribution based on a new attack GMVFool, while the outer minimization trains a viewpoint-invariant classifier by minimizing the expected loss over the worst-case adversarial viewpoint distributions. To further improve the generalization performance, a distribution sharing strategy is introduced leveraging the transferability of adversarial viewpoints across objects. Experiments validate the effectiveness of VIAT in improving the viewpoint robustness of various image classifiers based on the diversity of adversarial viewpoints generated by GMVFool.