# Addressing Topic Leakage in Cross-Topic Evaluation for Authorship Verification

arXiv: https://arxiv.org/abs/2407.19164 · HTML: https://mlsanity.com/paper/2407.19164/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 32 references that are in the map.
- First posted on arXiv 27 Jul 2024.

## Builds on (most cited first, 11 of 11)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases](https://mlsanity.com/paper/1909.03683/) (2019)
- [Towards Debiasing NLU Models from Unknown Biases](https://mlsanity.com/paper/2009.12303/) (2020)
- [Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary](https://mlsanity.com/paper/2010.00490/) (2020)
- [Same Author or Just Same Topic? Towards Content-Independent Style Representations](https://mlsanity.com/paper/2204.04907/) (2022)
- [Mind the Trade-off: Debiasing NLU Models without Degrading the In-distribution Performance](https://mlsanity.com/paper/2005.00315/) (2020)
- [The Topic Confusion Task: A Novel Scenario for Authorship Attribution](https://mlsanity.com/paper/2104.08530/) (2021)
- [O2D2: Out-Of-Distribution Detector to Capture Undecidable Trials in Authorship Verification](https://mlsanity.com/paper/2106.15825/) (2021)
- [Rethinking the Authorship Verification Experimental Setups](https://mlsanity.com/paper/2112.05125/) (2021)

## Built on by (most cited first, 2 of 2)

- [Capturing Classic Authorial Style in Long-Form Story Generation with GRPO Fine-Tuning](https://mlsanity.com/paper/2512.05747/) (2025)
- [Contrastive Learning for Authorship Verification](https://mlsanity.com/paper/2609.28471/) (2026)

## Abstract (arXiv)

Authorship verification (AV) aims to identify whether a pair of texts has the same author. We address the challenge of evaluating AV models' robustness against topic shifts. The conventional evaluation assumes minimal topic overlap between training and test data. However, we argue that there can still be topic leakage in test data, causing misleading model performance and unstable rankings. To address this, we propose an evaluation method called Heterogeneity-Informed Topic Sampling (HITS), which creates a smaller dataset with a heterogeneously distributed topic set. Our experimental results demonstrate that HITS-sampled datasets yield a more stable ranking of models across random seeds and evaluation splits. Our contributions include: 1. An analysis of causes and effects of topic leakage. 2. A demonstration of the HITS in reducing the effects of topic leakage, and 3. The Robust Authorship Verification bENchmark (RAVEN) that allows topic shortcut test to uncover AV models' reliance on topic-specific features.