# RaTE: a Reproducible automatic Taxonomy Evaluation by Filling the Gap

arXiv: https://arxiv.org/abs/2307.09706 · HTML: https://mlsanity.com/paper/2307.09706/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 30 references that are in the map.
- First posted on arXiv 19 Jul 2023.

## Builds on (most cited first, 8 of 8)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [ERNIE: Enhanced Representation through Knowledge Integration](https://mlsanity.com/paper/1904.09223/) (2019)
- [FinBERT: A Pretrained Language Model for Financial Communications](https://mlsanity.com/paper/2006.08097/) (2020)
- [Automated Phrase Mining from Massive Text Corpora](https://mlsanity.com/paper/1702.04457/) (2017)
- [TaxoGen: Unsupervised Topic Taxonomy Construction by Adaptive Term Embedding and Clustering](https://mlsanity.com/paper/1812.09551/) (2018)
- [Ultra-Fine Entity Typing with Weak Supervision from a Masked Language Model](https://mlsanity.com/paper/2106.04098/) (2021)
- [HiExpan: Task-Guided Taxonomy Construction by Hierarchical Tree Expansion](https://mlsanity.com/paper/1910.08194/) (2019)
- [CoRel: Seed-Guided Topical Taxonomy Construction by Concept Learning and Relation Transferring](https://mlsanity.com/paper/2010.06714/) (2020)

## Built on by (most cited first, 4 of 4)

- [Chain-of-Layer: Iteratively Prompting Large Language Models for Taxonomy Induction from Limited Examples](https://mlsanity.com/paper/2402.07386/) (2024)
- [Bridging the Evaluation Gap: Leveraging Large Language Models for Topic Model Evaluation](https://mlsanity.com/paper/2502.07352/) (2025)
- [Taxonomy Inference for Tabular Data Using Large Language Models](https://mlsanity.com/paper/2503.21810/) (2025)
- [Reference-Free Evaluation of Taxonomies](https://mlsanity.com/paper/2505.11470/) (2025)

## Abstract (arXiv)

Taxonomies are an essential knowledge representation, yet most studies on automatic taxonomy construction (ATC) resort to manual evaluation to score proposed algorithms. We argue that automatic taxonomy evaluation (ATE) is just as important as taxonomy construction. We propose RaTE, an automatic label-free taxonomy scoring procedure, which relies on a large pre-trained language model. We apply our evaluation procedure to three state-of-the-art ATC algorithms with which we built seven taxonomies from the Yelp domain, and show that 1) RaTE correlates well with human judgments and 2) artificially degrading a taxonomy leads to decreasing RaTE score.