# MIReAD: Simple Method for Learning High-quality Representations from Scientific Documents

arXiv: https://arxiv.org/abs/2305.04177 · HTML: https://mlsanity.com/paper/2305.04177/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 22 references that are in the map.
- First posted on arXiv 7 May 2023.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [Unsupervised Domain Clusters in Pretrained Language Models](https://mlsanity.com/paper/2004.02105/) (2020)
- [On the Use of ArXiv as a Dataset](https://mlsanity.com/paper/1905.00075/) (2019)
- [CiteWorth: Cite-Worthiness Detection for Improved Scientific Document Understanding](https://mlsanity.com/paper/2105.10912/) (2021)
- [Learning multi-scale functional representations of proteins from single-cell microscopy data](https://mlsanity.com/paper/2205.11676/) (2022)

## Built on by (most cited first, 2 of 2)

- [Improving Scientific Document Retrieval with Concept Coverage-based Query Set Generation](https://mlsanity.com/paper/2502.11181/) (2025)
- [Improving Scientific Document Retrieval with Academic Concept Index](https://mlsanity.com/paper/2601.00567/) (2026)

## Abstract (arXiv)

Learning semantically meaningful representations from scientific documents can facilitate academic literature search and improve performance of recommendation systems. Pre-trained language models have been shown to learn rich textual representations, yet they cannot provide powerful document-level representations for scientific articles. We propose MIReAD, a simple method that learns high-quality representations of scientific papers by fine-tuning transformer model to predict the target journal class based on the abstract. We train MIReAD on more than 500,000 PubMed and arXiv abstracts across over 2,000 journal classes. We show that MIReAD produces representations that can be used for similar papers retrieval, topic categorization and literature search. Our proposed approach outperforms six existing models for representation learning on scientific documents across four evaluation standards.