# exHarmony: Authorship and Citations for Benchmarking the Reviewer Assignment Problem

arXiv: https://arxiv.org/abs/2502.07683 · HTML: https://mlsanity.com/paper/2502.07683/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 47 references that are in the map.
- First posted on arXiv 11 Feb 2025.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and concepts](https://mlsanity.com/paper/2205.01833/) (2022)
- [On Strategyproof Conference Peer Review](https://mlsanity.com/paper/1806.06266/) (2018)
- [Shallow pooling for sparse labels](https://mlsanity.com/paper/2109.00062/) (2021)
- [A multi-label classification method using a hierarchical and transparent representation for paper-reviewer recommendation](https://mlsanity.com/paper/1912.08976/) (2019)
- [Generative Information Retrieval Evaluation](https://mlsanity.com/paper/2404.08137/) (2024)
- [Reviewer assignment problem: A scoping review](https://mlsanity.com/paper/2305.07887/) (2023)
- [Reviewer Assignment Problem: A Systematic Review of the Literature](https://mlsanity.com/paper/2304.00353/) (2023)

## Built on by (most cited first, 5 of 5)

- [PeerPrism: Peer Evaluation Expertise vs Review-writing AI](https://mlsanity.com/paper/2604.14513/) (2026)
- [Peerispect: Claim Verification in Scientific Peer Reviews](https://mlsanity.com/paper/2604.17667/) (2026)
- [PeeriScope: A Multi-Faceted Framework for Evaluating Peer Review Quality](https://mlsanity.com/paper/2604.24071/) (2026)
- [OmniReview: A Large-scale Benchmark and LLM-enhanced Framework for Realistic Reviewer Recommendation](https://mlsanity.com/paper/2602.08896/) (2026)
- [Judging a Review by its Cover: A Reliability Analysis of LLM-based Peer Review Evaluation Metrics](https://mlsanity.com/paper/2609.23264/) (2026)

## Abstract (arXiv)

The peer review process is crucial for ensuring the quality and reliability of scholarly work, yet assigning suitable reviewers remains a significant challenge. Traditional manual methods are labor-intensive and often ineffective, leading to nonconstructive or biased reviews. This paper introduces the exHarmony (eHarmony but for connecting experts to manuscripts) benchmark, designed to address these challenges by re-imagining the Reviewer Assignment Problem (RAP) as a retrieval task. Utilizing the extensive data from OpenAlex, we propose a novel approach that considers a host of signals from the authors, most similar experts, and the citation relations as potential indicators for a suitable reviewer for a manuscript. This approach allows us to develop a standard benchmark dataset for evaluating the reviewer assignment problem without needing explicit labels. We benchmark various methods, including traditional lexical matching, static neural embeddings, and contextualized neural embeddings, and introduce evaluation metrics that assess both relevance and diversity in the context of RAP. Our results indicate that while traditional methods perform reasonably well, contextualized embeddings trained on scholarly literature show the best performance. The findings underscore the importance of further research to enhance the diversity and effectiveness of reviewer assignments.