# Improving Pretrained Models for Zero-shot Multi-label Text Classification through Reinforced Label Hierarchy Reasoning

arXiv: https://arxiv.org/abs/2104.01666 · HTML: https://mlsanity.com/paper/2104.01666/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 31 references that are in the map.
- First posted on arXiv 4 Apr 2021.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach](https://mlsanity.com/paper/1909.00161/) (2019)
- [DeepPath: A Reinforcement Learning Method for Knowledge Graph Reasoning](https://mlsanity.com/paper/1707.06690/) (2017)
- [Multi-Stage Document Ranking with BERT](https://mlsanity.com/paper/1910.14424/) (2019)
- [HDLTex: Hierarchical Deep Learning for Text Classification](https://mlsanity.com/paper/1709.08267/) (2017)
- [Large-Scale Multi-Label Text Classification on EU Legislation](https://mlsanity.com/paper/1906.02192/) (2019)
- [Reinforcement Knowledge Graph Reasoning for Explainable Recommendation](https://mlsanity.com/paper/1906.05237/) (2019)
- [Understanding the Behaviors of BERT in Ranking](https://mlsanity.com/paper/1904.07531/) (2019)
- [An Empirical Study on Large-Scale Multi-Label Text Classification Including Few and Zero-Shot Labels](https://mlsanity.com/paper/2010.01653/) (2020)
- [Hierarchical Text Classification with Reinforced Label Assignment](https://mlsanity.com/paper/1908.10419/) (2019)
- [Learning to Learn and Predict: A Meta-Learning Approach for Multi-Label Classification](https://mlsanity.com/paper/1909.04176/) (2019)
- [Universal Text Representation from BERT: An Empirical Study](https://mlsanity.com/paper/1910.07973/) (2019)
- [LSHTC: A Benchmark for Large-Scale Text Classification](https://mlsanity.com/paper/1503.08581/) (2015)
- [Multi-label Few/Zero-shot Learning with Knowledge Aggregated from Multiple Label Graphs](https://mlsanity.com/paper/2010.07459/) (2020)
- [ML-Net: multi-label classification of biomedical texts with deep neural networks](https://mlsanity.com/paper/1811.05475/) (2018)

## Built on by (most cited first, 9 of 9)

- [CHiLS: Zero-Shot Image Classification with Hierarchical Label Sets](https://mlsanity.com/paper/2302.02551/) (2023)
- [Eliciting the Translation Ability of Large Language Models via Multilingual Finetuning with Translation Instructions](https://mlsanity.com/paper/2305.15083/) (2023)
- [QUEACO: Borrowing Treasures from Weakly-labeled Behavior Data for Query Attribute Value Extraction](https://mlsanity.com/paper/2108.08468/) (2021)
- [Open-Vocabulary Object Detection via Language Hierarchy](https://mlsanity.com/paper/2410.20371/) (2024)
- [RLET: A Reinforcement Learning Based Approach for Explainable QA with Entailment Trees](https://mlsanity.com/paper/2210.17095/) (2022)
- [A Simple and Effective Framework for Strict Zero-Shot Hierarchical Classification](https://mlsanity.com/paper/2305.15282/) (2023)
- [Learning an Artificial Language for Knowledge-Sharing in Multilingual Translation](https://mlsanity.com/paper/2211.01292/) (2022)
- [Leveraging Hierarchical Taxonomies in Prompt-based Continual Learning](https://mlsanity.com/paper/2410.04327/) (2024)
- [Are We Really Making Much Progress in Text Classification? A Comparative Review](https://mlsanity.com/paper/2204.03954/) (2022)

## Abstract (arXiv)

Exploiting label hierarchies has become a promising approach to tackling the zero-shot multi-label text classification (ZS-MTC) problem. Conventional methods aim to learn a matching model between text and labels, using a graph encoder to incorporate label hierarchies to obtain effective label representations citerios2018few. More recently, pretrained models like BERT citedevlin2018bert have been used to convert classification tasks into a textual entailment task citeyin-etal-2019-benchmarking. This approach is naturally suitable for the ZS-MTC task. However, pretrained models are underexplored in the existing work because they do not generate individual vector representations for text or labels, making it unintuitive to combine them with conventional graph encoding methods. In this paper, we explore to improve pretrained models with label hierarchies on the ZS-MTC task. We propose a Reinforced Label Hierarchy Reasoning (RLHR) approach to encourage interdependence among labels in the hierarchies during training. Meanwhile, to overcome the weakness of flat predictions, we design a rollback algorithm that can remove logical errors from predictions during inference. Experimental results on three real-life datasets show that our approach achieves better performance and outperforms previous non-pretrained methods on the ZS-MTC task.