# Who is a Better Matchmaker? Human vs. Algorithmic Judge Assignment in a High-Stakes Startup Competition

arXiv: https://arxiv.org/abs/2510.12692 · HTML: https://mlsanity.com/paper/2510.12692/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 51 references that are in the map.
- First posted on arXiv 14 Oct 2025.

## Builds on (most cited first, 22 of 22)

- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [SciRepEval: A Multi-Format Benchmark for Scientific Document Representations](https://mlsanity.com/paper/2211.13308/) (2022)
- [Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings](https://mlsanity.com/paper/2202.06671/) (2022)
- [Mitigating Manipulation in Peer Review via Randomized Reviewer Assignments](https://mlsanity.com/paper/2006.16437/) (2020)
- [PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Review](https://mlsanity.com/paper/1806.06237/) (2018)
- [Paper Matching with Local Fairness Constraints](https://mlsanity.com/paper/1905.11924/) (2019)
- [On Strategyproof Conference Peer Review](https://mlsanity.com/paper/1806.06266/) (2018)
- [An Improved Text Sentiment Classification Model Using TF-IDF and Next Word Negation](https://mlsanity.com/paper/1806.06407/) (2018)
- [PaRe: A Paper-Reviewer Matching Approach Using a Common Topic Space](https://mlsanity.com/paper/1909.11258/) (2019)
- [Representation learning for very short texts using weighted word embedding aggregation](https://mlsanity.com/paper/1607.00570/) (2016)
- [A Gold Standard Dataset for the Reviewer Assignment Problem](https://mlsanity.com/paper/2303.16750/) (2023)
- [No more Reviewer #2: Subverting Automatic Paper-Reviewer Assignment using Adversarial Learning](https://mlsanity.com/paper/2303.14443/) (2023)
- [Vulnerability of Text-Matching in ML/AI Conference Reviewer Assignments to Collusions](https://mlsanity.com/paper/2412.06606/) (2024)
- [Near-Optimal Reviewer Splitting in Two-Phase Paper Reviewing and Conference Experiment Design](https://mlsanity.com/paper/2108.06371/) (2021)
- [Strategyproofing Peer Assessment via Partitioning: The Price in Terms of Evaluators' Expertise](https://mlsanity.com/paper/2201.10631/) (2022)
- [A One-Size-Fits-All Approach to Improving Randomness in Paper Assignment](https://mlsanity.com/paper/2310.05995/) (2023)
- [Learning representations of learning representations](https://mlsanity.com/paper/2404.08403/) (2024)
- [Document Embedding for Scientific Articles: Efficacy of Word Embeddings vs TFIDF](https://mlsanity.com/paper/2107.05151/) (2021)
- [Identity Theft in AI Conference Peer Review](https://mlsanity.com/paper/2508.04024/) (2025)
- [The Devil is in the Details: Evaluating Limitations of Transformer-based Methods for Granular Tasks](https://mlsanity.com/paper/2011.01196/) (2020)
- [Enhancing Peer Review in Astronomy: A Machine Learning and Optimization Approach to Reviewer Assignments for ALMA](https://mlsanity.com/paper/2410.10009/) (2024)

## Abstract (arXiv)

There is growing interest in applying artificial intelligence (AI) to automate and support complex decision-making tasks. However, it remains unclear how algorithms compare to human judgment in contexts requiring semantic understanding and domain expertise. We examine this in the context of the judge assignment problem, matching submissions to suitably qualified judges. Specifically, we tackled this problem at the Harvard President's Innovation Challenge, the university's premier venture competition awarding over 500,000 to student and alumni startups. This represents a real-world environment where high-quality judge assignment is essential. We developed an AI-based judge-assignment algorithm, Hybrid Lexical-Semantic Similarity Ensemble (HLSE), and deployed it at the competition. We then evaluated its performance against human expert assignments using blinded match-quality scores from judges on 309 judge-venture pairs. Using a Mann-Whitney U statistic based test, we found no statistically significant difference in assignment quality between the two approaches (AUC=0.48, p=0.40); on average, algorithmic matches are rated 3.90 and manual matches 3.94 on a 5-point scale, where 5 indicates an excellent match. Furthermore, manual assignments that previously required a full week could be automated in several hours by the algorithm during deployment. These results demonstrate that HLSE achieves human-expert-level matching quality while offering greater scalability and efficiency, underscoring the potential of AI-driven solutions to support and enhance human decision-making for judge assignment in high-stakes settings.