Hawk: An Industrial-strength Multi-label Document Classifier
Not on the map. Why?
Abstract (arXiv)
There are a plethora of methods and algorithms that solve the classical multi-label document classification. However, when it comes to deployment and usage in an industry setting, most, if not all the contemporary approaches fail to address some of the vital aspects or requirements of an ideal solution: i. ability to operate on variable-length texts and rambling documents. ii. catastrophic forgetting problem. iii. modularity when it comes to online learning and updating the model. iv. ability to spotlight relevant text while producing the prediction, i.e. visualizing the predictions. v. ability to operate on imbalanced or skewed datasets. vi. scalability. The paper describes the significance of these problems in detail and proposes a unique neural network architecture that addresses the above problems. The proposed architecture views documents as a sequence of sentences and leverages sentence-level embeddings for input representation. A hydranet-like architecture is designed to have granular control over and improve the modularity, coupled with a weighted loss driving task-specific heads. In particular, two specific mechanisms are compared: Bi-LSTM and Transformer-based. The architecture is benchmarked on some of the popular benchmarking datasets such as Web of Science - 5763, Web of Science - 11967, BBC Sports, and BBC News datasets. The experimental results reveal that the proposed model outperforms the existing methods by a substantial margin. The ablation study includes comparisons of the impact of the attention mechanism and the application of weighted loss functions to train the task-specific heads in the hydranet.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 22 of its 37 references that are in the map. First posted on arXiv 15 Jan 2023.
Builds on
22 of 37 references.
References · 22
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
- 2020Longformer: The Long-Document Transformercited by 3,362
- 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
- 2014Convolutional Neural Networks for Sentence Classificationcited by 1,124
- 2018Universal Sentence Encodercited by 485
- 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
- 2017HDLTex: Hierarchical Deep Learning for Text Classificationcited by 105
- 2019Hierarchical Transformers for Long Document Classificationcited by 103
- 2017DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understandingcited by 71
- 2018Pseudo-Rehearsal: Achieving Deep Reinforcement Learning without Catastrophic Forgettingcited by 46
10 more
- 2014Scale-Invariant Convolutional Neural Networkscited by 40
- 2014Multi-scale Orderless Pooling of Deep Convolutional Activation Featurescited by 38
- 2021Sequence Length is a Domain: Length-based Overfitting in Transformer Modelscited by 34
- 2022Efficient Classification of Long Documents Using Transformerscited by 30
- 2016Sequential Short-Text Classification with Recurrent and Convolutional Neural Networkscited by 28
- 2019Message Passing Attention Networks for Document Understandingcited by 23
- 2019Rep the Set: Neural Networks for Learning Set Representationscited by 22
- 2022A Survey for Efficient Open Domain Question Answeringcited by 20
- 2021Comparative Study of Long Document Classificationcited by 12
- 2019Speeding up Word Mover's Distance and its variants via properties of distances between embeddingscited by 5
Built on by
Citing papers per year: 2023: 1.
1 paper.