mlsanityData to 14 Oct 2026

OmniCharacter: Towards Immersive Role-Playing Agents with Seamless Speech-Language Personality Interaction

arXiv 2505.20277 ↗First posted 26 May 2025cs.CL cs.CVcited by 14

On the map: LLM agents and tool use → Social simulation and multi-agent debate

Abstract (arXiv)

Role-Playing Agents (RPAs), benefiting from large language models, is an emerging interactive AI system that simulates roles or characters with diverse personalities. However, existing methods primarily focus on mimicking dialogues among roles in textual form, neglecting the role's voice traits (e.g., voice style and emotions) as playing a crucial effect in interaction, which tends to be more immersive experiences in realistic scenarios. Towards this goal, we propose OmniCharacter, a first seamless speech-language personality interaction model to achieve immersive RPAs with low latency. Specifically, OmniCharacter enables agents to consistently exhibit role-specific personality traits and vocal traits throughout the interaction, enabling a mixture of speech and language responses. To align the model with speech-language scenarios, we construct a dataset named OmniCharacter-10K, which involves more distinctive characters (20), richly contextualized multi-round dialogue (10K), and dynamic speech response (135K). Experimental results showcase that our method yields better responses in terms of both content and style compared to existing RPAs and mainstream speech-language models, with a response latency as low as 289ms. Code and dataset are available at https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/OmniCharacter.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 27 of its 42 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

27 of 42 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025PsyPlay: Personality-Infused Role-Playing Conversational Agents6 Feb
  2. 2024SocialBench: Sociality Evaluation of Role-Playing Conversational Agents20 Mar
  3. 2024CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation2 Jan
  4. 2023Character-LLM: A Trainable Agent for Role-Playing16 Oct
  5. 2023Generative Agents: Interactive Simulacra of Human Behavior7 Apr

Other references · 22

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  2. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  3. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  4. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  5. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
  6. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  7. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  8. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  9. 2024GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbotcited by 224
  10. 2023InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviewscited by 179
  11. 2018AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scalecited by 128
  12. 2023CAM++: A Fast and Efficient Network for Speaker Verification Using Context-Aware Maskingcited by 102
10 more
  1. 2024OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversationcited by 60
  2. 2024MMEvol: Empowering Multimodal Large Language Models with Evol-Instructcited by 42
  3. 2024CharacterBox: Evaluating the Role-Playing Capabilities of LLMs in Text-Based Virtual Worldscited by 38
  4. 2024SpeechGPT-Gen: Scaling Chain-of-Information Speech Generationcited by 37
  5. 2024DEEM: Diffusion Models Serve as the Eyes of Large Language Models for Image Perceptioncited by 32
  6. 2024Mitigating Hallucination in Fictional Character Role-Playcited by 18
  7. 2024Quantifying and Optimizing Global Faithfulness in Persona-driven Role-playingcited by 17
  8. 2025FisherTune: Fisher-Guided Robust Tuning of Vision Foundation Models for Domain Generalized Segmentationcited by 16
  9. 2024BEYOND DIALOGUE: A Profile-Dialogue Alignment Framework Towards General Role-Playing Language Modelcited by 13
  10. 2025Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understandingcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 10.

14 papers.

Sort:

By area:

  1. 2026Token Reduction via Local and Global Contexts Optimization for Efficient Video Large Language Modelscited by 5
  2. 2025SpeechRole: A Large-Scale Dataset and Benchmark for Evaluating Speech Role-Playing Agentscited by 4
  3. 2026Rethinking Role-Playing Evaluation: Anonymous Benchmarking and a Systematic Study of Personality Effectscited by 3
  4. 2025Act-Adaptive Margin: Dynamically Calibrating Reward Models for Subjective Ambiguitycited by 2
  5. 2025VoxRole: A Comprehensive Benchmark for Evaluating Speech-Based Role-Playing Agentscited by 2
  6. 2026DREAM: LLM-based Dynamic Role-playing via Event-Aware Memory Graphcited by 2
  7. 2025RevoNAD: Reflective Evolutionary Exploration for Neural Architecture Designcited by 1
  8. 2026VITA-QinYu: Expressive Spoken Language Model for Role-Playing and Singingcited by 1
  9. 2026From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agentscited by 1
  10. 2026Bagpiper-TTS: Natural Language Guided Universal Speech Synthesiscited by 1
  11. 2026MERRY: Semantically Decoupled Evaluation of Multimodal Emotional and Role Consistencies of Role-Playing Agentscited by 0
  12. 2026Character Beyond Speech: Leveraging Role-Playing Evaluation in Audio Large Language Models via Reinforcement Learningcited by 0
2 more
  1. 2026Ex-Omni-2D: Expressive Omni-Modal Dialogue Models with Native Visual Presencecited by 0
  2. 2026From Memory to Behavior: A Behavior-Aware Role-Playing Framework for Social Media Influencerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Multi-Task Role-Playing Agent Capable of Imitating Character Linguistic Styles
  2. 2026Role-Playing Agents Driven by Large Language Models: Current Status, Challenges, and Future Trends
  3. 2025Test-Time-Matching: Decouple Personality, Memory, and Linguistic Style in LLM-based Role-Playing Language Agent
  4. 2024From Persona to Personalization: A Survey on Role-Playing Language Agents
  5. 2024Crafting Customisable Characters with LLMs: A Persona-Driven Role-Playing Agent Framework
  6. 2026Deep Persona: A Psychologically Grounded Architecture and Evaluation Framework for Role-Playing Agents and Simulations
  7. 2024Speaker Verification in Agent-Generated Conversations
  8. 2026AdaMARP: An Adaptive Multi-Agent Interaction Framework for General Immersive Role-Playing