# Towards Visual Taxonomy Expansion

arXiv: https://arxiv.org/abs/2309.06105 · HTML: https://mlsanity.com/paper/2309.06105/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 35 references that are in the map.
- First posted on arXiv 12 Sep 2023.

## Builds on (most cited first, 17 of 17)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Junction Tree Variational Autoencoder for Molecular Graph Generation](https://mlsanity.com/paper/1802.04364/) (2018)
- [TaxoExpan: Self-supervised Taxonomy Expansion with Position-Enhanced Graph Neural Network](https://mlsanity.com/paper/2001.09522/) (2020)
- [STEAM: Self-Supervised Taxonomy Expansion with Mini-Paths](https://mlsanity.com/paper/2006.10217/) (2020)
- [Improving Hypernymy Detection with an Integrated Path-based and Distributional Method](https://mlsanity.com/paper/1603.06076/) (2016)
- [SetExpan: Corpus-Based Set Expansion via Context Feature Selection and Rank Ensemble](https://mlsanity.com/paper/1910.08192/) (2019)
- [Temperature as Uncertainty in Contrastive Learning](https://mlsanity.com/paper/2110.04403/) (2021)
- [Taxonomy Completion via Triplet Matching Network](https://mlsanity.com/paper/2101.01896/) (2021)
- [TaxoEnrich: Self-Supervised Taxonomy Completion via Structure-Semantic Representations](https://mlsanity.com/paper/2202.04887/) (2022)
- [Enhancing Taxonomy Completion with Concept Generation via Fusing Relational Representations](https://mlsanity.com/paper/2106.02974/) (2021)
- [End-to-End Reinforcement Learning for Automatic Taxonomy Induction](https://mlsanity.com/paper/1805.04044/) (2018)
- [HyperExpan: Taxonomy Expansion with Hyperbolic Representation Learning](https://mlsanity.com/paper/2109.10500/) (2021)
- [Who Should Go First? A Self-Supervised Concept Sorting Model for Improving Taxonomy Expansion](https://mlsanity.com/paper/2104.03682/) (2021)
- [Learning What You Need from What You Did: Product Taxonomy Expansion with User Behaviors Supervision](https://mlsanity.com/paper/2203.14921/) (2022)

## Abstract (arXiv)

Taxonomy expansion task is essential in organizing the ever-increasing volume of new concepts into existing taxonomies. Most existing methods focus exclusively on using textual semantics, leading to an inability to generalize to unseen terms and the "Prototypical Hypernym Problem." In this paper, we propose Visual Taxonomy Expansion (VTE), introducing visual features into the taxonomy expansion task. We propose a textual hypernymy learning task and a visual prototype learning task to cluster textual and visual semantics. In addition to the tasks on respective modalities, we introduce a hyper-proto constraint that integrates textual and visual semantics to produce fine-grained visual semantics. Our method is evaluated on two datasets, where we obtain compelling results. Specifically, on the Chinese taxonomy dataset, our method significantly improves accuracy by 8.75 %. Additionally, our approach performs better than ChatGPT on the Chinese taxonomy dataset.