# Evaluating Role-Consistency in LLMs for Counselor Training

arXiv: https://arxiv.org/abs/2601.08892 · HTML: https://mlsanity.com/paper/2601.08892/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 31 references that are in the map.
- First posted on arXiv 13 Jan 2026.

## Builds on (most cited first, 13 of 13)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models](https://mlsanity.com/paper/2306.11698/) (2023)
- [LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Models](https://mlsanity.com/paper/2305.13711/) (2023)
- [Roleplay-doh: Enabling Domain-Experts to Create LLM-simulated Patients via Eliciting and Adhering to Principles](https://mlsanity.com/paper/2407.00870/) (2024)
- [Prompted LLMs as Chatbot Modules for Long Open-domain Conversation](https://mlsanity.com/paper/2305.04533/) (2023)
- [Observing Dialogue in Therapy: Categorizing and Forecasting Behavioral Codes](https://mlsanity.com/paper/1907.00326/) (2019)
- [Leveraging Large Language Model as Simulated Patients for Clinical Education](https://mlsanity.com/paper/2404.13066/) (2024)
- [Conversation Model Fine-Tuning for Classifying Client Utterances in Counseling Dialogues](https://mlsanity.com/paper/1904.00350/) (2019)
- [Persona-Based Conversational AI: State of the Art and Challenges](https://mlsanity.com/paper/2212.03699/) (2022)

## Built on by (most cited first, 1 of 1)

- [Transition-Matrix Regularization for Next Dialogue Act Prediction in Counselling Conversations](https://mlsanity.com/paper/2604.18539/) (2026)

## Abstract (arXiv)

The rise of online counseling services has highlighted the need for effective training methods for future counselors. This paper extends research on VirCo, a Virtual Client for Online Counseling, designed to complement traditional role-playing methods in academic training by simulating realistic client interactions. Building on previous work, we introduce a new dataset incorporating adversarial attacks to test the ability of large language models (LLMs) to maintain their assigned roles (role-consistency). The study focuses on evaluating the role consistency and coherence of the Vicuna model's responses, comparing these findings with earlier research. Additionally, we assess and compare various open-source LLMs for their performance in sustaining role consistency during virtual client interactions. Our contributions include creating an adversarial dataset, evaluating conversation coherence and persona consistency, and providing a comparative analysis of different LLMs.