# Weakly-Supervised Scientific Document Classification via Retrieval-Augmented Multi-Stage Training

arXiv: https://arxiv.org/abs/2306.07193 · HTML: https://mlsanity.com/paper/2306.07193/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation.
- Builds on 30 of its 48 references that are in the map.
- First posted on arXiv 12 Jun 2023.

## Placed in

- Retrieval-augmented generation / [Generation and context integration](https://mlsanity.com/rag/generation-and-context-integration/)

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval](https://mlsanity.com/paper/2108.05540/) (2021)
- [On the Use of ArXiv as a Dataset](https://mlsanity.com/paper/1905.00075/) (2019)
- [WRENCH: A Comprehensive Benchmark for Weak Supervision](https://mlsanity.com/paper/2109.11377/) (2021)
- [BOND: BERT-Assisted Open-Domain Named Entity Recognition with Distant Supervision](https://mlsanity.com/paper/2006.15509/) (2020)
- [Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approach](https://mlsanity.com/paper/2010.07835/) (2020)
- [FBNETGEN: Task-aware GNN-based fMRI Analysis via Functional Brain Network Generation](https://mlsanity.com/paper/2205.12465/) (2022)
- [Weakly-Supervised Neural Text Classification](https://mlsanity.com/paper/1809.01478/) (2018)
- [X-Class: Text Classification with Extremely Weak Supervision](https://mlsanity.com/paper/2010.12794/) (2020)
- [COCO-DR: Combating Distribution Shifts in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust Learning](https://mlsanity.com/paper/2210.15212/) (2022)
- [Query Expansion with Locally-Trained Word Embeddings](https://mlsanity.com/paper/1605.07891/) (2016)
- [The Effect of Metadata on Scientific Literature Tagging: A Cross-Field Cross-Model Study](https://mlsanity.com/paper/2302.03341/) (2023)
- [Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understanding](https://mlsanity.com/paper/2305.14232/) (2023)
- [Neighborhood-Regularized Self-Training for Learning with Few Labels](https://mlsanity.com/paper/2301.03726/) (2023)
- [ReSel: N-ary Relation Extraction from Scientific Text and Tables by Learning to Retrieve and Select](https://mlsanity.com/paper/2210.14427/) (2022)
- [PTGB: Pre-Train Graph Neural Networks for Brain Network Analysis](https://mlsanity.com/paper/2305.14376/) (2023)
- [MotifClass: Weakly Supervised Text Classification with Higher-order Metadata Information](https://mlsanity.com/paper/2111.04022/) (2021)
- [How Can Graph Neural Networks Help Document Retrieval: A Case Study on CORD19 with Concept Map Generation](https://mlsanity.com/paper/2201.04672/) (2022)
- [Seed-Guided Topic Discovery with Out-of-Vocabulary Seeds](https://mlsanity.com/paper/2205.01845/) (2022)
- [A Benchmark on Extremely Weakly Supervised Text Classification: Reconcile Seed Matching and Prompting Approaches](https://mlsanity.com/paper/2305.12749/) (2023)
- [Adaptive Multi-view Rule Discovery for Weakly-Supervised Compatible Products Prediction](https://mlsanity.com/paper/2206.13749/) (2022)
- [FastClass: A Time-Efficient Approach to Weakly-Supervised Text Classification](https://mlsanity.com/paper/2212.05506/) (2022)

## Built on by (most cited first, 8 of 8)

- [ToolQA: A Dataset for LLM Question Answering with External Tools](https://mlsanity.com/paper/2306.13304/) (2023)
- [RAM-EHR: Retrieval Augmentation Meets Clinical Predictions on Electronic Health Records](https://mlsanity.com/paper/2403.00815/) (2024)
- [Knowledge-Infused Prompting: Assessing and Advancing Clinical Text Data Generation with Large Language Models](https://mlsanity.com/paper/2311.00287/) (2023)
- [ARL2: Aligning Retrievers for Black-box Large Language Models via Self-guided Adaptive Relevance Labeling](https://mlsanity.com/paper/2402.13542/) (2024)
- [HiGen: Hierarchy-Aware Sequence Generation for Hierarchical Text Classification](https://mlsanity.com/paper/2402.01696/) (2024)
- [Open Visual Knowledge Extraction via Relation-Oriented Multimodality Model Prompting](https://mlsanity.com/paper/2310.18804/) (2023)
- [Efficient Few-shot Learning for Multi-label Classification of Scientific Documents with Many Classes](https://mlsanity.com/paper/2410.05770/) (2024)
- [Artificial Intuition: Efficient Classification of Scientific Abstracts](https://mlsanity.com/paper/2407.06093/) (2024)

## Abstract (arXiv)

Scientific document classification is a critical task for a wide range of applications, but the cost of obtaining massive amounts of human-labeled data can be prohibitive. To address this challenge, we propose a weakly-supervised approach for scientific document classification using label names only. In scientific domains, label names often include domain-specific concepts that may not appear in the document corpus, making it difficult to match labels and documents precisely. To tackle this issue, we propose WANDER, which leverages dense retrieval to perform matching in the embedding space to capture the semantics of label names. We further design the label name expansion module to enrich the label name representations. Lastly, a self-training step is used to refine the predictions. The experiments on three datasets show that WANDER outperforms the best baseline by 11.9% on average. Our code will be published at https://github.com/ritaranx/wander.