# On the Challenges of Open World Recognitionunder Shifting Visual Domains

arXiv: https://arxiv.org/abs/2107.04461 · HTML: https://mlsanity.com/paper/2107.04461/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 42 references that are in the map.
- First posted on arXiv 9 Jul 2021.

## Builds on (most cited first, 30 of 30)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Domain Generalization by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1903.06864/) (2019)
- [Self-Challenging Improves Cross-Domain Generalization](https://mlsanity.com/paper/2007.02454/) (2020)
- [Learning to Learn Single Domain Generalization](https://mlsanity.com/paper/2003.13216/) (2020)
- [Towards Open World Recognition](https://mlsanity.com/paper/1412.5687/) (2014)
- [Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networks](https://mlsanity.com/paper/1812.07252/) (2018)
- [Analyzing and Improving Representations with the Soft Nearest Neighbor Loss](https://mlsanity.com/paper/1902.01889/) (2019)
- [On the Effectiveness of Image Rotation for Open Set Domain Adaptation](https://mlsanity.com/paper/2007.12360/) (2020)
- [Towards Recognizing Unseen Categories in Unseen Domains](https://mlsanity.com/paper/2007.12256/) (2020)
- [Incremental Adversarial Domain Adaptation for Continually Changing Environments](https://mlsanity.com/paper/1712.07436/) (2017)
- [Adversarial Feature Augmentation for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1711.08561/) (2017)
- [Multi-Task Domain Adaptation for Deep Learning of Instance Grasping from Simulation](https://mlsanity.com/paper/1710.06422/) (2017)
- [Class-Incremental Domain Adaptation](https://mlsanity.com/paper/2008.01389/) (2020)
- [VR-Goggles for Robots: Real-to-sim Domain Adaptation for Visual Control](https://mlsanity.com/paper/1802.00265/) (2018)
- [AdaGraph: Unifying Predictive and Continuous Domain Adaptation through Graphs](https://mlsanity.com/paper/1903.07062/) (2019)
- [Kitting in the Wild through Online Domain Adaptation](https://mlsanity.com/paper/1807.01028/) (2018)
- [Addressing Appearance Change in Outdoor Robotics with Adversarial Domain Adaptation](https://mlsanity.com/paper/1703.01461/) (2017)
- [Self-Supervised Sim-to-Real Adaptation for Visual Robotic Manipulation](https://mlsanity.com/paper/1910.09470/) (2019)
- [Robust Place Categorization with Deep Domain Generalization](https://mlsanity.com/paper/1805.12048/) (2018)
- [Unsupervised Domain Adaptation through Inter-modal Rotation for RGB-D Object Recognition](https://mlsanity.com/paper/2004.10016/) (2020)
- [Knowledge is Never Enough: Towards Web Aided Deep Open World Recognition](https://mlsanity.com/paper/1906.01258/) (2019)
- [DEDUCE: Diverse scEne Detection methods in Unseen Challenging Environments](https://mlsanity.com/paper/1908.00191/) (2019)
- [Recognizing Objects In-the-wild: Where Do We Stand?](https://mlsanity.com/paper/1709.05862/) (2017)
- [Boosting Deep Open World Recognition by Clustering](https://mlsanity.com/paper/2004.13849/) (2020)
- [Online Open World Recognition](https://mlsanity.com/paper/1604.02275/) (2016)
- [Learning Deep NBNN Representations for Robust Place Categorization](https://mlsanity.com/paper/1702.07898/) (2017)
- [Adaptive Deep Learning through Visual Domain Localization](https://mlsanity.com/paper/1802.08833/) (2018)
- [KittingBot: A Mobile Manipulation Robot for Collaborative Kitting in Automotive Logistics](https://mlsanity.com/paper/1809.05380/) (2018)

## Built on by (most cited first, 1 of 1)

- [Contrastive Learning for Cross-Domain Open World Recognition](https://mlsanity.com/paper/2203.09257/) (2022)

## Abstract (arXiv)

Robotic visual systems operating in the wild must act in unconstrained scenarios, under different environmental conditions while facing a variety of semantic concepts, including unknown ones. To this end, recent works tried to empower visual object recognition methods with the capability to i) detect unseen concepts and ii) extended their knowledge over time, as images of new semantic classes arrive. This setting, called Open World Recognition (OWR), has the goal to produce systems capable of breaking the semantic limits present in the initial training set. However, this training set imposes to the system not only its own semantic limits, but also environmental ones, due to its bias toward certain acquisition conditions that do not necessarily reflect the high variability of the real-world. This discrepancy between training and test distribution is called domain-shift. This work investigates whether OWR algorithms are effective under domain-shift, presenting the first benchmark setup for assessing fairly the performances of OWR algorithms, with and without domain-shift. We then use this benchmark to conduct analyses in various scenarios, showing how existing OWR algorithms indeed suffer a severe performance degradation when train and test distributions differ. Our analysis shows that this degradation is only slightly mitigated by coupling OWR with domain generalization techniques, indicating that the mere plug-and-play of existing algorithms is not enough to recognize new and unknown categories in unseen domains. Our results clearly point toward open issues and future research directions, that need to be investigated for building robot visual systems able to function reliably under these challenging yet very real conditions. Code available at https://github.com/DarioFontanel/OWR-VisualDomains