# Towards Visual Semantics

arXiv: https://arxiv.org/abs/2104.12379 · HTML: https://mlsanity.com/paper/2104.12379/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 66 references that are in the map.
- First posted on arXiv 26 Apr 2021.

## Builds on (most cited first, 20 of 20)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [Lifelong Learning with Dynamically Expandable Networks](https://mlsanity.com/paper/1708.01547/) (2017)
- [Towards Open Set Deep Networks](https://mlsanity.com/paper/1511.06233/) (2015)
- [A Survey of Embodied AI: From Simulators to Research Tasks](https://mlsanity.com/paper/2103.04918/) (2021)
- [YFCC100M: The New Data in Multimedia Research](https://mlsanity.com/paper/1503.01817/) (2015)
- [Towards Open World Recognition](https://mlsanity.com/paper/1412.5687/) (2014)
- [Continual Unsupervised Representation Learning](https://mlsanity.com/paper/1910.14481/) (2019)
- [A Survey on Learning to Hash](https://mlsanity.com/paper/1606.00185/) (2016)
- [Generative replay with feedback connections as a general strategy for continual learning](https://mlsanity.com/paper/1809.10635/) (2018)
- [The Extreme Value Machine](https://mlsanity.com/paper/1506.06112/) (2015)
- [Task Agnostic Continual Learning Using Online Variational Bayes](https://mlsanity.com/paper/1803.10123/) (2018)
- [Hashing for Similarity Search: A Survey](https://mlsanity.com/paper/1408.2927/) (2014)
- [Continual egocentric object recognition](https://mlsanity.com/paper/1912.05029/) (2019)

## Built on by (most cited first, 12 of 12)

- [Aligning Visual and Lexical Semantics](https://mlsanity.com/paper/2212.06629/) (2022)
- [A semantics-driven methodology for high-quality image annotation](https://mlsanity.com/paper/2307.14119/) (2023)
- [Object Recognition as Classification via Visual Properties](https://mlsanity.com/paper/2112.10531/) (2021)
- [Visual Ground Truth Construction as Faceted Classification](https://mlsanity.com/paper/2202.08512/) (2022)
- [Brainish: Formalizing A Multimodal Language for Intelligence and Consciousness](https://mlsanity.com/paper/2205.00001/) (2022)
- [Classifying concepts via visual properties](https://mlsanity.com/paper/2105.09422/) (2021)
- [Building a visual semantics aware object hierarchy](https://mlsanity.com/paper/2202.13021/) (2022)
- [Representation Heterogeneity](https://mlsanity.com/paper/2207.01091/) (2022)
- [Incremental Image Labeling via Iterative Refinement](https://mlsanity.com/paper/2304.08989/) (2023)
- [Egocentric Hierarchical Visual Semantics](https://mlsanity.com/paper/2305.05422/) (2023)
- [Lifelong Personal Context Recognition](https://mlsanity.com/paper/2205.10123/) (2022)
- [From Knowledge Organization to Knowledge Representation and Back](https://mlsanity.com/paper/2401.11753/) (2024)

## Abstract (arXiv)

Lexical Semantics is concerned with how words encode mental representations of the world, i.e., concepts . We call this type of concepts, classification concepts . In this paper, we focus on Visual Semantics , namely on how humans build concepts representing what they perceive visually. We call this second type of concepts, substance concepts . As shown in the paper, these two types of concepts are different and, furthermore, the mapping between them is many-to-many. In this paper we provide a theory and an algorithm for how to build substance concepts which are in a one-to-one correspondence with classifications concepts, thus paving the way to the seamless integration between natural language descriptions and visual perception. This work builds upon three main intuitions: (i) substance concepts are modeled as visual objects , namely sequences of similar frames, as perceived in multiple encounters ; (ii) substance concepts are organized into a visual subsumption hierarchy based on the notions of Genus and Differentia ; (iii) the human feedback is exploited not to name objects, but, rather, to align the hierarchy of substance concepts with that of classification concepts. The learning algorithm is implemented for the base case of a hierarchy of depth two. The experiments, though preliminary, show that the algorithm manages to acquire the notions of Genus and Differentia with reasonable accuracy, this despite seeing a small number of examples and receiving supervision on a fraction of them.