# Beyond One-Hot-Encoding: Injecting Semantics to Drive Image Classifiers

arXiv: https://arxiv.org/abs/2308.00607 · HTML: https://mlsanity.com/paper/2308.00607/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 41 references that are in the map.
- First posted on arXiv 1 Aug 2023.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [RISE: Randomized Input Sampling for Explanation of Black-box Models](https://mlsanity.com/paper/1806.07421/) (2018)
- [Evaluation of Output Embeddings for Fine-Grained Image Classification](https://mlsanity.com/paper/1409.8403/) (2014)
- [Explainable artificial intelligence (XAI) in deep learning-based medical image analysis](https://mlsanity.com/paper/2107.10912/) (2021)
- [Latent Embeddings for Zero-shot Classification](https://mlsanity.com/paper/1603.08895/) (2016)
- [Making Better Mistakes: Leveraging Class Hierarchies with Deep Networks](https://mlsanity.com/paper/1912.09393/) (2019)
- [Hierarchy-based Image Embeddings for Semantic Image Retrieval](https://mlsanity.com/paper/1809.09924/) (2018)
- [Learning Hierarchy Aware Features for Reducing Mistake Severity](https://mlsanity.com/paper/2207.12646/) (2022)
- [Improving Interpretability of Deep Neural Networks with Semantic Information](https://mlsanity.com/paper/1703.04096/) (2017)

## Built on by (most cited first, 1 of 1)

- [Improving deep learning with prior knowledge and cognitive models: A survey on enhancing explainability, adversarial robustness and zero-shot learning](https://mlsanity.com/paper/2403.07078/) (2024)

## Abstract (arXiv)

Images are loaded with semantic information that pertains to real-world ontologies: dog breeds share mammalian similarities, food pictures are often depicted in domestic environments, and so on. However, when training machine learning models for image classification, the relative similarities amongst object classes are commonly paired with one-hot-encoded labels. According to this logic, if an image is labelled as 'spoon', then 'tea-spoon' and 'shark' are equally wrong in terms of training loss. To overcome this limitation, we explore the integration of additional goals that reflect ontological and semantic knowledge, improving model interpretability and trustworthiness. We suggest a generic approach that allows to derive an additional loss term starting from any kind of semantic information about the classification label. First, we show how to apply our approach to ontologies and word embeddings, and discuss how the resulting information can drive a supervised learning process. Second, we use our semantically enriched loss to train image classifiers, and analyse the trade-offs between accuracy, mistake severity, and learned internal representations. Finally, we discuss how this approach can be further exploited in terms of explainability and adversarial robustness. Code repository: https://github.com/S1M0N38/semantic-encodings