# Can Authorship Attribution Models Distinguish Speakers in Speech Transcripts?

arXiv: https://arxiv.org/abs/2311.07564 · HTML: https://mlsanity.com/paper/2311.07564/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 36 references that are in the map.
- First posted on arXiv 13 Nov 2023.

## Builds on (most cited first, 11 of 11)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learning](https://mlsanity.com/paper/2008.03648/) (2020)
- [Same Author or Just Same Topic? Towards Content-Independent Style Representations](https://mlsanity.com/paper/2204.04907/) (2022)
- [Speaker Anonymization Using X-vector and Neural Waveform Models](https://mlsanity.com/paper/1905.13561/) (2019)
- [Explainable Authorship Verification in Social Media via Attention-based Similarity Learning](https://mlsanity.com/paper/1910.08144/) (2019)
- [A Deep Metric Learning Approach to Account Linking](https://mlsanity.com/paper/2105.07263/) (2021)
- [Idiosyncratic but not Arbitrary: Learning Idiolects in Online Registers Reveals Distinctive yet Consistent Individual Styles](https://mlsanity.com/paper/2109.03158/) (2021)
- [Can Authorship Representation Learning Capture Stylistic Features?](https://mlsanity.com/paper/2308.11490/) (2023)
- [Learning Stylometric Representations for Authorship Analysis](https://mlsanity.com/paper/1606.01219/) (2016)
- [PART: Pre-trained Authorship Representation Transformer](https://mlsanity.com/paper/2209.15373/) (2022)
- [HANSEN: Human and AI Spoken Text Benchmark for Authorship Analysis](https://mlsanity.com/paper/2310.16746/) (2023)

## Built on by (most cited first, 7 of 7)

- [Speaker Verification in Agent-Generated Conversations](https://mlsanity.com/paper/2405.10150/) (2024)
- [Improving Quotation Attribution with Fictional Character Embeddings](https://mlsanity.com/paper/2406.11368/) (2024)
- [Content Anonymization for Privacy in Long-form Audio](https://mlsanity.com/paper/2510.12780/) (2025)
- [The Impact of Automatic Speech Transcription on Speaker Attribution](https://mlsanity.com/paper/2507.08660/) (2025)
- [A stylometric analysis of speaker attribution from speech transcripts](https://mlsanity.com/paper/2512.13667/) (2025)
- [Authorship Verification of Transcribed German-Language Videos](https://mlsanity.com/paper/2607.29168/) (2026)
- [Back in Style: A Sociolinguistic Approach to Authoring and Measuring Persona Fidelity in User Simulation](https://mlsanity.com/paper/2610.10988/) (2026)

## Abstract (arXiv)

Authorship verification is the task of determining if two distinct writing samples share the same author and is typically concerned with the attribution of written text. In this paper, we explore the attribution of transcribed speech, which poses novel challenges. The main challenge is that many stylistic features, such as punctuation and capitalization, are not informative in this setting. On the other hand, transcribed speech exhibits other patterns, such as filler words and backchannels (e.g., 'um', 'uh-huh'), which may be characteristic of different speakers. We propose a new benchmark for speaker attribution focused on human-transcribed conversational speech transcripts. To limit spurious associations of speakers with topic, we employ both conversation prompts and speakers participating in the same conversation to construct verification trials of varying difficulties. We establish the state of the art on this new benchmark by comparing a suite of neural and non-neural baselines, finding that although written text attribution models achieve surprisingly good performance in certain settings, they perform markedly worse as conversational topic is increasingly controlled. We present analyses of the impact of transcription style on performance as well as the ability of fine-tuning on speech transcripts to improve performance.