Graph Regularized Encoder Training for Extreme Classification
Not on the map. Why?
Abstract (arXiv)
Deep extreme classification (XC) aims to train an encoder architecture and an accompanying classifier architecture to tag a data point with the most relevant subset of labels from a very large universe of labels. XC applications in ranking, recommendation and tagging routinely encounter tail labels for which the amount of training data is exceedingly small. Graph convolutional networks (GCN) present a convenient but computationally expensive way to leverage task metadata and enhance model accuracies in these settings. This paper formally establishes that in several use cases, the steep computational cost of GCNs is entirely avoidable by replacing GCNs with non-GCN architectures. The paper notices that in these settings, it is much more effective to use graph data to regularize encoder training than to implement a GCN. Based on these insights, an alternative paradigm RAMEN is presented to utilize graph metadata in XC settings that offers significant performance boosts with zero increase in inference computational costs. RAMEN scales to datasets with up to 1M labels and offers prediction accuracy up to 15% higher on benchmark datasets than state of the art methods, including those that use graph metadata to train GCNs. RAMEN also offers 10% higher accuracy over the best baseline on a proprietary recommendation dataset sourced from click logs of a popular search engine. Code for RAMEN will be released publicly.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 41 of its 71 references that are in the map. First posted on arXiv 28 Feb 2024.
Builds on
41 of 71 references.
References · 41
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
- 2017Inductive Representation Learning on Large Graphscited by 6,320
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
- 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
- 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
- 2020LightGCN: Simplifying and Powering Graph Convolution Network for Recommendationcited by 1,139
- 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 849
- 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 695
- 2019GraphSAINT: Graph Sampling Based Inductive Learning Methodcited by 618
- 2018FastGCN: Fast Learning with Graph Convolutional Networks via Importance Samplingcited by 584
29 more
- 2019Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networkscited by 541
- 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 413
- 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 350
- 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
- 2020Sparse, Dense, and Attentional Representations for Text Retrievalcited by 214
- 2018Adaptive Sampling Towards Fast Graph Representation Learningcited by 180
- 2019Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networkscited by 177
- 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
- 2021GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graphcited by 143
- 2016DiSMEC - Distributed Sparse Machines for Extreme Multi-label Classificationcited by 71
- 2022Knowledge Graph Contrastive Learning for Recommendationcited by 70
- 2021Fast Multi-Resolution Transformer Fine-tuning for Extreme Multi-label Text Classificationcited by 67
- 2019Bonsai -- Diverse and Shallow Trees for Extreme Multi-label Classificationcited by 61
- 2021LightXML: Transformer with Dynamic Negative Sampling for High-Performance Extreme Multi-label Text Classificationcited by 61
- 2021TextGNN: Improving Text Encoder via Graph Neural Network in Sponsored Searchcited by 54
- 2018A no-regret generalization of hierarchical softmax to extreme multi-label classificationcited by 52
- 2021ECLARE: Extreme Classification with Label Graph Correlationscited by 38
- 2021DeepXML: A Deep Extreme Multi-Label Learning Framework Applied to Short Text Documentscited by 37
- 2019Extreme Classification in Log Memory using Count-Min Sketch: A Case Study of Amazon Search with 50M Productscited by 36
- 2021DECAF: Deep Extreme Classification with Label Featurescited by 33
- 2022NGAME: Negative Mining-aware Mini-batching for Extreme Classificationcited by 32
- 2018Stochastic Negative Mining for Learning with Large Output Spacescited by 29
- 2019Extreme Multi-Label Legal Text Classification: A case study in EU Legislationcited by 26
- 2020Pretrained Generalized Autoregressive Model with Adaptive Probabilistic Label Clusters for Extreme Multi-label Text Classificationcited by 25
- 2017Deep Extreme Multi-label Learningcited by 24
- 2022CascadeXML: Rethinking Transformers for End-to-end Multi-resolution Training in Extreme Multi-label Classificationcited by 12
- 2021Disentangling Sampling and Labeling Bias for Learning in Large-Output Spacescited by 9
- 2023PINA: Leveraging Side Information in eXtreme Multi-label Classification via Predicted Instance Neighborhood Aggregationcited by 7
- 2023Dual-Encoders for Extreme Multi-Label Classificationcited by 7
Built on by
Citing papers per year: 2024: 1.
1 paper.