# A Transformer-based Autoregressive Decoder Architecture for Hierarchical Text Classification

arXiv: https://arxiv.org/abs/2501.13598 · HTML: https://mlsanity.com/paper/2501.13598/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 45 references that are in the map.
- First posted on arXiv 23 Jan 2025.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [HDLTex: Hierarchical Deep Learning for Text Classification](https://mlsanity.com/paper/1709.08267/) (2017)
- [Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluations](https://mlsanity.com/paper/2306.04618/) (2023)
- [SGM: Sequence Generation Model for Multi-label Classification](https://mlsanity.com/paper/1806.04822/) (2018)
- [A Survey on Text Classification: From Shallow to Deep Learning](https://mlsanity.com/paper/2008.00364/) (2020)
- [Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? A Study on Several Typical Tasks](https://mlsanity.com/paper/2305.05862/) (2023)
- [Incorporating Hierarchy into Text Encoder: a Contrastive Learning Approach for Hierarchical Text Classification](https://mlsanity.com/paper/2203.03825/) (2022)
- [Open, Closed, or Small Language Models for Text Classification?](https://mlsanity.com/paper/2308.10092/) (2023)
- [Pushing The Limit of LLM Capacity for Text Classification](https://mlsanity.com/paper/2402.07470/) (2024)
- [HTCInfoMax: A Global Model for Hierarchical Text Classification via Information Maximization](https://mlsanity.com/paper/2104.05220/) (2021)
- [Scheduling Techniques for Liver Segmentation: ReduceLRonPlateau Vs OneCycleLR](https://mlsanity.com/paper/2202.06373/) (2022)
- [Exploiting Global and Local Hierarchies for Hierarchical Text Classification](https://mlsanity.com/paper/2205.02613/) (2022)
- [Constrained Sequence-to-Tree Generation for Hierarchical Text Classification](https://mlsanity.com/paper/2204.00811/) (2022)
- [Recommendations for Item Set Completion: On the Semantics of Item Co-Occurrence With Data Sparsity, Input Size, and Input Modalities](https://mlsanity.com/paper/2105.04376/) (2021)
- [AI Insights: A Case Study on Utilizing ChatGPT Intelligence for Research Paper Analysis](https://mlsanity.com/paper/2403.03293/) (2024)

## Abstract (arXiv)

Recent approaches in hierarchical text classification (HTC) rely on the capabilities of a pre-trained transformer model and exploit the label semantics and a graph encoder for the label hierarchy. In this paper, we introduce an effective hierarchical text classifier RADAr (Transformer-based Autoregressive Decoder Architecture) that is based only on an off-the-shelf RoBERTa transformer to process the input and a custom autoregressive decoder with two decoder layers for generating the classification output. Thus, unlike existing approaches for HTC, the encoder of RADAr has no explicit encoding of the label hierarchy and the decoder solely relies on the label sequences of the samples observed during training. We demonstrate on three benchmark datasets that RADAr achieves results competitive to the state of the art with less training and inference time. Our model consistently performs better when organizing the label sequences from children to parents versus the inverse, as done in existing HTC approaches. Our experiments show that neither the label semantics nor an explicit graph encoder for the hierarchy is needed. This has strong practical implications for HTC as the architecture has fewer requirements and provides a speed-up by a factor of 2 at inference time. Moreover, training a separate decoder from scratch in conjunction with fine-tuning the encoder allows future researchers and practitioners to exchange the encoder part as new models arise. The source code is available at https://github.com/yousef-younes/RADAr.