# Specialized Document Embeddings for Aspect-based Similarity of Research Papers

arXiv: https://arxiv.org/abs/2203.14541 · HTML: https://mlsanity.com/paper/2203.14541/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 83 references that are in the map.
- First posted on arXiv 28 Mar 2022.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [LINE: Large-scale Information Network Embedding](https://mlsanity.com/paper/1503.03578/) (2015)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [Efficient Natural Language Response Suggestion for Smart Reply](https://mlsanity.com/paper/1705.00652/) (2017)
- [Not Enough Data? Deep Learning to the Rescue!](https://mlsanity.com/paper/1911.03118/) (2019)
- [ANN-Benchmarks: A Benchmarking Tool for Approximate Nearest Neighbor Algorithms](https://mlsanity.com/paper/1807.05614/) (2018)
- [Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings](https://mlsanity.com/paper/2202.06671/) (2022)
- [From Word to Sense Embeddings: A Survey on Vector Representations of Meaning](https://mlsanity.com/paper/1805.04032/) (2018)
- [AxCell: Automatic Extraction of Results from Machine Learning Papers](https://mlsanity.com/paper/2004.14356/) (2020)
- [Recurrent Chunking Mechanisms for Long-Text Machine Reading Comprehension](https://mlsanity.com/paper/2005.08056/) (2020)
- [Adversarial Training for Aspect-Based Sentiment Analysis with BERT](https://mlsanity.com/paper/2001.11316/) (2020)
- [Aspect-based Document Similarity for Research Papers](https://mlsanity.com/paper/2010.06395/) (2020)
- [Semantic Specialisation of Distributional Word Vector Spaces using Monolingual and Cross-Lingual Constraints](https://mlsanity.com/paper/1706.00374/) (2017)
- [Are Neural Language Models Good Plagiarists? A Benchmark for Neural Paraphrase Detection](https://mlsanity.com/paper/2103.12450/) (2021)
- [Identifying Machine-Paraphrased Plagiarism](https://mlsanity.com/paper/2103.11909/) (2021)
- [Bursting Scientific Filter Bubbles: Boosting Innovation via Novel Author Discovery](https://mlsanity.com/paper/2108.05669/) (2021)
- [De-Conflated Semantic Representations](https://mlsanity.com/paper/1608.01961/) (2016)
- [hyperdoc2vec: Distributed Representations of Hypertext Documents](https://mlsanity.com/paper/1805.03793/) (2018)
- [VitaLITy: Promoting Serendipitous Discovery of Academic Literature with Transformers & Visual Analytics](https://mlsanity.com/paper/2108.03366/) (2021)
- [Testing the Generalization of Neural Language Models for COVID-19 Misinformation Detection](https://mlsanity.com/paper/2111.07819/) (2021)
- [Do Multi-Sense Embeddings Improve Natural Language Understanding?](https://mlsanity.com/paper/1506.01070/) (2015)
- [Text Data Augmentation: Towards better detection of spear-phishing emails](https://mlsanity.com/paper/2007.02033/) (2020)

## Built on by (most cited first, 12 of 12)

- [SciRepEval: A Multi-Format Benchmark for Scientific Document Representations](https://mlsanity.com/paper/2211.13308/) (2022)
- [A Domain-adaptive Pre-training Approach for Language Bias Detection in News](https://mlsanity.com/paper/2205.10773/) (2022)
- [Interpretable Text Embeddings and Text Similarity Explanation: A Survey](https://mlsanity.com/paper/2502.14862/) (2025)
- [MIR: Methodology Inspiration Retrieval for Scientific Research Problems](https://mlsanity.com/paper/2506.00249/) (2025)
- [Don't Reinvent the Wheel: Efficient Instruction-Following Text Embedding based on Guided Space Transformation](https://mlsanity.com/paper/2505.24754/) (2025)
- [AspectCSE: Sentence Embeddings for Aspect-based Semantic Textual Similarity Using Contrastive Learning and Structured Knowledge](https://mlsanity.com/paper/2307.07851/) (2023)
- [Paraphrase Types for Generation and Detection](https://mlsanity.com/paper/2310.14863/) (2023)
- [FLeW: Facet-Level and Adaptive Weighted Representation Learning of Scientific Documents](https://mlsanity.com/paper/2509.07531/) (2025)
- [SemCSE-Multi: Multifaceted and Decodable Embeddings for Aspect-Specific and Interpretable Scientific Domain Mapping](https://mlsanity.com/paper/2510.11599/) (2025)
- [CoCon: A Data Set on Combined Contextualized Research Artifact Use](https://mlsanity.com/paper/2303.15193/) (2023)
- [Enriching Social Science Research via Survey Item Linking](https://mlsanity.com/paper/2412.15831/) (2024)
- [The Critical Role of Aspects in Measuring Document Similarity](https://mlsanity.com/paper/2601.03435/) (2026)

## Abstract (arXiv)

Document embeddings and similarity measures underpin content-based recommender systems, whereby a document is commonly represented as a single generic embedding. However, similarity computed on single vector representations provides only one perspective on document similarity that ignores which aspects make two documents alike. To address this limitation, aspect-based similarity measures have been developed using document segmentation or pairwise multi-class document classification. While segmentation harms the document coherence, the pairwise classification approach scales poorly to large scale corpora. In this paper, we treat aspect-based similarity as a classical vector similarity problem in aspect-specific embedding spaces. We represent a document not as a single generic embedding but as multiple specialized embeddings. Our approach avoids document segmentation and scales linearly w.r.t.the corpus size. In an empirical study, we use the Papers with Code corpus containing 157,606 research papers and consider the task, method, and dataset of the respective research papers as their aspects. We compare and analyze three generic document embeddings, six specialized document embeddings and a pairwise classification baseline in the context of research paper recommendations. As generic document embeddings, we consider FastText, SciBERT, and SPECTER. To compute the specialized document embeddings, we compare three alternative methods inspired by retrofitting, fine-tuning, and Siamese networks. In our experiments, Siamese SciBERT achieved the highest scores. Additional analyses indicate an implicit bias of the generic document embeddings towards the dataset aspect and against the method aspect of each research paper. Our approach of aspect-based document embeddings mitigates potential risks arising from implicit biases by making them explicit.