# Speaker Verification in Agent-Generated Conversations

arXiv: https://arxiv.org/abs/2405.10150 · HTML: https://mlsanity.com/paper/2405.10150/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 20 of its 52 references that are in the map.
- First posted on arXiv 16 May 2024.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 20 of 20)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://mlsanity.com/paper/1801.07243/) (2018)
- [Character-LLM: A Trainable Agent for Role-Playing](https://mlsanity.com/paper/2310.10158/) (2023)
- [Generalized End-to-End Loss for Speaker Verification](https://mlsanity.com/paper/1710.10467/) (2017)
- [Beyond Goldfish Memory: Long-Term Open-Domain Conversation](https://mlsanity.com/paper/2107.07567/) (2021)
- [TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents](https://mlsanity.com/paper/1901.08149/) (2019)
- [CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation](https://mlsanity.com/paper/2401.01275/) (2024)
- [Same Author or Just Same Topic? Towards Content-Independent Style Representations](https://mlsanity.com/paper/2204.04907/) (2022)
- [CharacterChat: Learning towards Conversational AI with Personalized Social Support](https://mlsanity.com/paper/2308.10278/) (2023)
- [Learning Interpretable Style Embeddings via Prompting LLMs](https://mlsanity.com/paper/2305.12696/) (2023)
- [Is ChatGPT a Good Personality Recognizer? A Preliminary Study](https://mlsanity.com/paper/2307.03952/) (2023)
- [Can Authorship Representation Learning Capture Stylistic Features?](https://mlsanity.com/paper/2308.11490/) (2023)
- [How Far Are LLMs from Believable AI? A Benchmark for Evaluating the Believability of Human Behavior Simulation](https://mlsanity.com/paper/2312.17115/) (2023)
- [Can Authorship Attribution Models Distinguish Speakers in Speech Transcripts?](https://mlsanity.com/paper/2311.07564/) (2023)
- [Persona Authentication through Generative Dialogue](https://mlsanity.com/paper/2110.12949/) (2021)
- [FDN: Finite Difference Network with Hierarchical Convolutional Features for Text-independent Speaker Verification](https://mlsanity.com/paper/2108.07974/) (2021)

## Built on by (most cited first, 4 of 4)

- [Consistent Client Simulation for Motivational Interviewing-based Counseling](https://mlsanity.com/paper/2502.02802/) (2025)
- [CAMI: A Counselor Agent Supporting Motivational Interviewing through State Inference and Topic Exploration](https://mlsanity.com/paper/2502.02807/) (2025)
- [Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectives](https://mlsanity.com/paper/2506.09656/) (2025)
- [A Concise Agent is Less Expert: Revealing Side Effects of Using Style Features on Conversational Agents](https://mlsanity.com/paper/2601.10809/) (2026)

## Abstract (arXiv)

The recent success of large language models (LLMs) has attracted widespread interest to develop role-playing conversational agents personalized to the characteristics and styles of different speakers to enhance their abilities to perform both general and special purpose dialogue tasks. However, the ability to personalize the generated utterances to speakers, whether conducted by human or LLM, has not been well studied. To bridge this gap, our study introduces a novel evaluation challenge: speaker verification in agent-generated conversations, which aimed to verify whether two sets of utterances originate from the same speaker. To this end, we assemble a large dataset collection encompassing thousands of speakers and their utterances. We also develop and evaluate speaker verification models under experiment setups. We further utilize the speaker verification models to evaluate the personalization abilities of LLM-based role-playing models. Comprehensive experiments suggest that the current role-playing models fail in accurately mimicking speakers, primarily due to their inherent linguistic characteristics.