# Metadata-Induced Contrastive Learning for Zero-Shot Multi-Label Text Classification

arXiv: https://arxiv.org/abs/2202.05932 · HTML: https://mlsanity.com/paper/2202.05932/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 96 references that are in the map.
- First posted on arXiv 11 Feb 2022.

## Builds on (most cited first, 30 of 41)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [Heterogeneous Graph Attention Network](https://mlsanity.com/paper/1903.07293/) (2019)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [A Survey of Data Augmentation Approaches for NLP](https://mlsanity.com/paper/2105.03075/) (2021)
- [Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach](https://mlsanity.com/paper/1909.00161/) (2019)
- [DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations](https://mlsanity.com/paper/2006.03659/) (2020)
- [ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer](https://mlsanity.com/paper/2105.11741/) (2021)
- [Multi-Stage Document Ranking with BERT](https://mlsanity.com/paper/1910.14424/) (2019)
- [CLEAR: Contrastive Learning for Sentence Representation](https://mlsanity.com/paper/2012.15466/) (2020)
- [Large-Scale Multi-Label Text Classification on EU Legislation](https://mlsanity.com/paper/1906.02192/) (2019)
- [AttentionXML: Label Tree-based Attention-Aware Deep Model for High-Performance Extreme Multi-Label Text Classification](https://mlsanity.com/paper/1811.01727/) (2018)
- [Weakly-Supervised Neural Text Classification](https://mlsanity.com/paper/1809.01478/) (2018)
- [Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipeline](https://mlsanity.com/paper/2101.08751/) (2021)
- [X-Class: Text Classification with Extremely Weak Supervision](https://mlsanity.com/paper/2010.12794/) (2020)
- [A Web-scale system for scientific knowledge exploration](https://mlsanity.com/paper/1805.12216/) (2018)
- [ECLARE: Extreme Classification with Label Graph Correlations](https://mlsanity.com/paper/2108.00261/) (2021)
- [Universal Representation Learning of Knowledge Bases by Jointly Embedding Instances and Ontological Concepts](https://mlsanity.com/paper/2103.08115/) (2021)
- [Extreme Classification in Log Memory using Count-Min Sketch: A Case Study of Amazon Search with 50M Products](https://mlsanity.com/paper/1910.13830/) (2019)
- [Text Classification Using Label Names Only: A Language Model Self-Training Approach](https://mlsanity.com/paper/2010.07245/) (2020)
- [Description Based Text Classification with Reinforcement Learning](https://mlsanity.com/paper/2002.03067/) (2020)

## Built on by (most cited first, 10 of 10)

- [Large Language Models on Graphs: A Comprehensive Survey](https://mlsanity.com/paper/2312.02783/) (2023)
- [Can we Soft Prompt LLMs for Graph Learning Tasks?](https://mlsanity.com/paper/2402.10359/) (2024)
- [Trial2Vec: Zero-Shot Clinical Trial Document Similarity Search using Self-Supervision](https://mlsanity.com/paper/2206.14719/) (2022)
- [The Effect of Metadata on Scientific Literature Tagging: A Cross-Field Cross-Model Study](https://mlsanity.com/paper/2302.03341/) (2023)
- [Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understanding](https://mlsanity.com/paper/2305.14232/) (2023)
- [CascadeXML: Rethinking Transformers for End-to-end Multi-resolution Training in Extreme Multi-label Classification](https://mlsanity.com/paper/2211.00640/) (2022)
- [SemSup-XC: Semantic Supervision for Zero and Few-shot Extreme Classification](https://mlsanity.com/paper/2301.11309/) (2023)
- [Weakly Supervised Multi-Label Classification of Full-Text Scientific Papers](https://mlsanity.com/paper/2306.14003/) (2023)
- [Towards Domain Specification of Embedding Models in Medicine](https://mlsanity.com/paper/2507.19407/) (2025)
- [Using Large Language Models to Automate Category and Trend Analysis of Scientific Articles: An Application in Ophthalmology](https://mlsanity.com/paper/2308.16688/) (2023)

## Abstract (arXiv)

Large-scale multi-label text classification (LMTC) aims to associate a document with its relevant labels from a large candidate set. Most existing LMTC approaches rely on massive human-annotated training data, which are often costly to obtain and suffer from a long-tailed label distribution (i.e., many labels occur only a few times in the training set). In this paper, we study LMTC under the zero-shot setting, which does not require any annotated documents with labels and only relies on label surface names and descriptions. To train a classifier that calculates the similarity score between a document and a label, we propose a novel metadata-induced contrastive learning (MICoL) method. Different from previous text-based contrastive learning techniques, MICoL exploits document metadata (e.g., authors, venues, and references of research papers), which are widely available on the Web, to derive similar document-document pairs. Experimental results on two large-scale datasets show that: (1) MICoL significantly outperforms strong zero-shot text classification and contrastive learning baselines; (2) MICoL is on par with the state-of-the-art supervised metadata-aware LMTC method trained on 10K-200K labeled documents; and (3) MICoL tends to predict more infrequent labels than supervised methods, thus alleviates the deteriorated performance on long-tailed labels.