# Encoding Multi-Domain Scientific Papers by Ensembling Multiple CLS Tokens

arXiv: https://arxiv.org/abs/2309.04333 · HTML: https://mlsanity.com/paper/2309.04333/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 25 references that are in the map.
- First posted on arXiv 8 Sep 2023.

## Builds on (most cited first, 12 of 12)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings](https://mlsanity.com/paper/2202.06671/) (2022)
- [Content-Based Citation Recommendation](https://mlsanity.com/paper/1802.08301/) (2018)
- [Citation Recommendation: Approaches and Datasets](https://mlsanity.com/paper/2002.06961/) (2020)
- [Multi-Vector Models with Textual Guidance for Fine-Grained Scientific Document Similarity](https://mlsanity.com/paper/2111.08366/) (2021)
- [Aspect-based Document Similarity for Research Papers](https://mlsanity.com/paper/2010.06395/) (2020)
- [Multi-CLS BERT: An Efficient Alternative to Traditional Ensembling](https://mlsanity.com/paper/2210.05043/) (2022)
- [A multi-label classification method using a hierarchical and transparent representation for paper-reviewer recommendation](https://mlsanity.com/paper/1912.08976/) (2019)
- [HybridCite: A Hybrid Model for Context-Aware Citation Recommendation](https://mlsanity.com/paper/2002.06406/) (2020)

## Built on by (most cited first, 2 of 2)

- [HLM-Cite: Hybrid Language Model Workflow for Text-based Scientific Citation Prediction](https://mlsanity.com/paper/2410.09112/) (2024)
- [KEMP-PIP: A Feature-Fusion Based Approach for Pro-inflammatory Peptide Prediction](https://mlsanity.com/paper/2602.20198/) (2026)

## Abstract (arXiv)

Many useful tasks on scientific documents, such as topic classification and citation prediction, involve corpora that span multiple scientific domains. Typically, such tasks are accomplished by representing the text with a vector embedding obtained from a Transformer's single CLS token. In this paper, we argue that using multiple CLS tokens could make a Transformer better specialize to multiple scientific domains. We present Multi2SPE: it encourages each of multiple CLS tokens to learn diverse ways of aggregating token embeddings, then sums them up together to create a single vector representation. We also propose our new multi-domain benchmark, Multi-SciDocs, to test scientific paper vector encoders under multi-domain settings. We show that Multi2SPE reduces error by up to 25 percent in multi-domain citation prediction, while requiring only a negligible amount of computation in addition to one BERT forward pass.