# Egocentric Hierarchical Visual Semantics

arXiv: https://arxiv.org/abs/2305.05422 · HTML: https://mlsanity.com/paper/2305.05422/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 36 references that are in the map.
- First posted on arXiv 9 May 2023.

## Builds on (most cited first, 15 of 15)

- [From Recognition to Cognition: Visual Commonsense Reasoning](https://mlsanity.com/paper/1811.10830/) (2018)
- [A Corpus for Reasoning About Natural Language Grounded in Photographs](https://mlsanity.com/paper/1811.00491/) (2018)
- [Recent Advances in Open Set Recognition: A Survey](https://mlsanity.com/paper/1811.08581/) (2018)
- [Towards Open World Recognition](https://mlsanity.com/paper/1412.5687/) (2014)
- [Neural Prototype Trees for Interpretable Fine-grained Image Recognition](https://mlsanity.com/paper/2012.02046/) (2020)
- [Making Better Mistakes: Leveraging Class Hierarchies with Deep Networks](https://mlsanity.com/paper/1912.09393/) (2019)
- [The Extreme Value Machine](https://mlsanity.com/paper/1506.06112/) (2015)
- [Open Vocabulary Scene Parsing](https://mlsanity.com/paper/1703.08769/) (2017)
- [Hierarchical Novelty Detection for Visual Object Recognition](https://mlsanity.com/paper/1804.00722/) (2018)
- [Continual egocentric object recognition](https://mlsanity.com/paper/1912.05029/) (2019)
- [Towards Visual Semantics](https://mlsanity.com/paper/2104.12379/) (2021)
- [Aligning Visual and Lexical Semantics](https://mlsanity.com/paper/2212.06629/) (2022)
- [Online Open World Recognition](https://mlsanity.com/paper/1604.02275/) (2016)
- [Open-World Visual Recognition Using Knowledge Graphs](https://mlsanity.com/paper/1708.08310/) (2017)
- [Vocabulary-informed Extreme Value Learning](https://mlsanity.com/paper/1705.09887/) (2017)

## Built on by (most cited first, 3 of 3)

- [Interpretability is in the Mind of the Beholder: A Causal Framework for Human-interpretable Representation Learning](https://mlsanity.com/paper/2309.07742/) (2023)
- [A semantics-driven methodology for high-quality image annotation](https://mlsanity.com/paper/2307.14119/) (2023)
- [What can Computer Vision learn from Ranganathan?](https://mlsanity.com/paper/2601.22634/) (2026)

## Abstract (arXiv)

We are interested in aligning how people think about objects and what machines perceive, meaning by this the fact that object recognition, as performed by a machine, should follow a process which resembles that followed by humans when thinking of an object associated with a certain concept. The ultimate goal is to build systems which can meaningfully interact with their users, describing what they perceive in the users' own terms. As from the field of Lexical Semantics, humans organize the meaning of words in hierarchies where the meaning of, e.g., a noun, is defined in terms of the meaning of a more general noun, its genus, and of one or more differentiating properties, its differentia. The main tenet of this paper is that object recognition should implement a hierarchical process which follows the hierarchical semantic structure used to define the meaning of words. We achieve this goal by implementing an algorithm which, for any object, recursively recognizes its visual genus and its visual differentia. In other words, the recognition of an object is decomposed in a sequence of steps where the locally relevant visual features are recognized. This paper presents the algorithm and a first evaluation.