PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models
Not on the map. Why?
Abstract (arXiv)
Recent advances in duplex speech models have enabled natural, low-latency speech-to-speech interactions. However, existing models are restricted to a fixed role and voice, limiting their ability to support structured, role-driven real-world applications and personalized interactions. In this work, we introduce PersonaPlex, a duplex conversational speech model that incorporates hybrid system prompts, combining role conditioning with text prompts and voice cloning with speech samples. PersonaPlex is trained on a large-scale synthetic dataset of paired prompts and user-agent conversations, generated with open-source large language models (LLM) and text-to-speech (TTS) models. To evaluate role conditioning in real-world settings, we extend the Full-Duplex-Bench benchmark beyond a single assistant role to multi-role customer service scenarios. Experiments show that PersonaPlex achieves strong role-conditioned behavior, voice-conditioned speech, and natural conversational responsiveness, surpassing state-of-the-art duplex speech models and hybrid large language model-based speech systems in role adherence, speaker similarity, latency, and naturalness.
Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Builds on 23 of its 33 references that are in the map. First posted on arXiv 14 Jan 2026.
Builds on
23 of 33 references.
References · 23
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2025Qwen3 Technical Reportcited by 8,195
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
- 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
- 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
- 2025Qwen2.5-Omni Technical Reportcited by 704
- 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
- 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
- 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
- 2024GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbotcited by 218
- 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 204
- 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 192
11 more
- 2024Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLMcited by 135
- 2024Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilitiescited by 104
- 2025Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilitiescited by 94
- 2023Libriheavy: a 50,000 hours ASR corpus with punctuation casing and contextcited by 84
- 2025LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesiscited by 75
- 2024Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agentscited by 69
- 2024OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversationcited by 57
- 2025SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Modelcited by 32
- 2025URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Modelscited by 29
- 2025Streaming Sequence-to-Sequence Learning with Delayed Streams Modelingcited by 24
- 2025VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Modelscited by 16
Built on by
Citing papers per year: 2026: 57.
57 papers.
Sort:
By area:
- 2026MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Modelscited by 11
- 2026Multi-Faceted Interactivity Alignment in Full-Duplex Speech Modelscited by 9
- 2026DuplexCascade: Full-Duplex Speech-to-Speech Dialogue with VAD-Free Cascaded ASR-LLM-TTS Pipeline and Micro-Turn Optimizationcited by 7
- 2026Raon-Speech Technical Reportcited by 4
- 2026Instruct-FD: Can Your Full-Duplex Speech System Follow Turn-Taking Instructions?cited by 4
- 2026VoxServe: Streaming-Centric Serving System for Speech Language Modelscited by 3
- 2026ASPIRin: Action Space Projection for Interactivity-Optimized Reinforcement Learning in Full-Duplex Speech Language Modelscited by 3
- 2026DuplexChat: Constructing Speaker-Separated Full-Duplex Dialogue Speech at Scale for Spoken Dialogue Language Modelingcited by 3
- 2026VoiceChat-TTS: A Low-Latency Continuous Speech Synthesis Model for Interactive Agentscited by 3
- 2026NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilitiescited by 3
- 2026DualTurn: Learning Turn-Taking from Dual-Channel Generative Speech Pretrainingcited by 2
- 2026MM-tau-p2: Persona-Adaptive Prompting for Robust Multi-Modal Agent Evaluation in Dual-Control Settingscited by 2
38 more
- 2026Overcoming State Inertia in Full-Duplex Spoken Language Models via Activation Steeringcited by 2
- 2026Cocktail-Talker: Multi-Speaker Dialog Modeling in Noisy Social Environments with Turn Action GRPOcited by 2
- 2026M3-DuplexBench: A Multi-Turn, Multilingual, Multidomain Benchmark for Full-Duplex Spoken Dialogue Modelscited by 2
- 2026MP-Bench: Evaluating Voice Agents as a Multiparty Conversation Participantcited by 2
- 2026SpokenUS: A Spoken User Simulator for Task-Oriented Dialoguecited by 1
- 2026TiCo: Time-Controllable Spoken Dialogue Modelcited by 1
- 2026RelayS2S: A Dual-Path Speculative Generation for Real-Time Dialoguecited by 1
- 2026VideoFDB: Evaluating Full-Duplex Vision-Speech Capabilities in Conversational Agentscited by 1
- 2026DyaPlex: Full-Duplex Speech-Motion Model for Dyadic Interactioncited by 1
- 2026X2-Turn: Frame-Synchronous Dual-Head Modeling for Joint Streaming ASR and Turn State Predictioncited by 1
- 2026DuplexSpeechBench-IFEval: Evaluating Implicit Instruction Following in Full-Duplex Voice Agentscited by 1
- 2026Noise Adaptive Streaming Audio-Visual Speech Token Enhancement for Robust Full-Duplex Spoken Dialogue Modelscited by 1
- 2026DuplexDrama: A Synthesized Dialogue Dataset with Scenarios, Full-Duplex Behaviors, Expressive Speech, and Sound Eventscited by 1
- 2026Enabling Streaming User Transcription in Full-Duplex Speech-to-Speech Modelscited by 1
- 2026A frontend-backend architecture for tool calls in full-duplex speech modelscited by 1
- 2026Full-Duplex Speech Models Take the Floor When Asked, Not When Neededcited by 1
- 2026Sommelier: Scalable Open Multi-turn Audio Pre-processing for Full-duplex Speech Language Modelscited by 0
- 2026Micro Language Models Enable Instant Responsescited by 0
- 2026PolySpeech-100: A Large-Scale Benchmark for Speech Understanding Across 100+ Languages and Dialectscited by 0
- 2026IRAF: Interference-Resilient Adaptive Fusion for Noise-Robust End-to-End Full-Duplex Spoken Dialogue Systemscited by 0
- 2026FacePlex: Toward Natural Full-Duplex Conversational Avatarscited by 0
- 2026Noisy-Channel Minimum Bayes Risk Decodingcited by 0
- 2026Low-Latency Turn-Taking via Context-Aware Preface Generation in a Real-World Dialogue Robotcited by 0
- 2026DuplexGen: Adaptive Synthesis of Human-AI Turn-Taking Dialoguescited by 0
- 2026Agentic-DuplexGen: Decoupling Content, Timing, and Acoustics for Synthetic Dialogue Speechcited by 0
- 2026Does Listening Matter? Backchanneling and Nodding in AI Clonecited by 0
- 2026Enabling Proactive Spoken Turns via a Generalized Style-Aware Full-Duplex Frameworkcited by 0
- 2026KABURI-TTS: Phoneme-Keyed Activity-conditioned Bi-channel Utterance Rendering for Interactioncited by 0
- 2026DuplexJail: Safety Alignment Breaks Under Spoken Interruption in Full-Duplex Modelscited by 0
- 2026RetroThinker: Enabling Retrospective Thinking in Speech LLMscited by 0
- 2026SteerDuplex: Steerable Duplex Speech Dialogue Modelscited by 0
- 2026Continue, Adapt, or Yield: In-Turn Adaptation to Overlapping Speech in Full-Duplex Agentscited by 0
- 2026Causal Analysis and Mitigation of Spurious Onsets in Full-Duplex Speech LLMscited by 0
- 2026RoleBreak: Benchmarking Long-Horizon Role-Playing Robustness in Spoken Dialoguecited by 0
- 2026When Should a Failing Robot Ask? Initiating Corrective Human-Robot Dialogue from Audited Sensor Evidencecited by 0
- 2026Neither Silence nor Overlap Is Failure: Intent-Conditioned Evaluation of Turn-Taking in Full-Duplex Spoken Dialogue Modelscited by 0
- 2026Controlling Backchannels in Streamable Full-duplex Modelscited by 0
- 2026Distributional Metrics for Evaluating Spoken Conversational Systemscited by 0
50 most cited of 57 All 57 (JSON)
- 2026EgoVoice: Proactive Spoken Assistance from Egocentric Multimodal Streams8 Oct 2026
- 2026Prompts versus Rules: Auditing and Controlling Speech Naturalness Behaviors in Voice User Simulators7 Oct 2026
- 2026DuplexAct-Bench: Broadening Full-Duplex Speech Evaluation toward Proactive Interaction across Diverse Behavioral Requirements30 Sep 2026
- 2026EvolvingAvatar: Interactive 3D Head Generation That Adapts as Conversations Unfold28 Sep 2026
- 2026DuplexCadence: Exact State and Execution from a Speech Model's Declared Timelines28 Sep 2026
- 2026Context Spanning: A Communication Framework for Full-Duplex Speech Models and External LLM Backends27 Sep 2026
- 2026IndicFDB: Benchmarking Full-Duplex Voice Agents across Indian Languages25 Sep 2026
- 2026Controlling Backchannels in Streamable Full-duplex Models24 Sep 2026
- 2026Neither Silence nor Overlap Is Failure: Intent-Conditioned Evaluation of Turn-Taking in Full-Duplex Spoken Dialogue Models23 Sep 2026
- 2026Distributional Metrics for Evaluating Spoken Conversational Systems21 Sep 2026
- 2026NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilities18 Sep 2026
- 2026When Should a Failing Robot Ask? Initiating Corrective Human-Robot Dialogue from Audited Sensor Evidence18 Sep 2026
38 more
- 2026Full-Duplex Speech Models Take the Floor When Asked, Not When Needed17 Sep 2026
- 2026A frontend-backend architecture for tool calls in full-duplex speech models16 Sep 2026
- 2026RoleBreak: Benchmarking Long-Horizon Role-Playing Robustness in Spoken Dialogue15 Sep 2026
- 2026Enabling Streaming User Transcription in Full-Duplex Speech-to-Speech Models14 Sep 2026
- 2026Causal Analysis and Mitigation of Spurious Onsets in Full-Duplex Speech LLMs11 Sep 2026
- 2026Continue, Adapt, or Yield: In-Turn Adaptation to Overlapping Speech in Full-Duplex Agents11 Sep 2026
- 2026MP-Bench: Evaluating Voice Agents as a Multiparty Conversation Participant11 Sep 2026
- 2026DuplexDrama: A Synthesized Dialogue Dataset with Scenarios, Full-Duplex Behaviors, Expressive Speech, and Sound Events11 Sep 2026
- 2026SteerDuplex: Steerable Duplex Speech Dialogue Models11 Sep 2026
- 2026RetroThinker: Enabling Retrospective Thinking in Speech LLMs10 Sep 2026
- 2026DuplexJail: Safety Alignment Breaks Under Spoken Interruption in Full-Duplex Models8 Sep 2026
- 2026Noise Adaptive Streaming Audio-Visual Speech Token Enhancement for Robust Full-Duplex Spoken Dialogue Models8 Sep 2026
- 2026KABURI-TTS: Phoneme-Keyed Activity-conditioned Bi-channel Utterance Rendering for Interaction7 Sep 2026
- 2026DuplexSpeechBench-IFEval: Evaluating Implicit Instruction Following in Full-Duplex Voice Agents3 Sep 2026
- 2026Does Listening Matter? Backchanneling and Nodding in AI Clone20 Aug 2026
- 2026Agentic-DuplexGen: Decoupling Content, Timing, and Acoustics for Synthetic Dialogue Speech17 Aug 2026
- 2026VoiceChat-TTS: A Low-Latency Continuous Speech Synthesis Model for Interactive Agents13 Aug 2026
- 2026X2-Turn: Frame-Synchronous Dual-Head Modeling for Joint Streaming ASR and Turn State Prediction11 Aug 2026
- 2026Enabling Proactive Spoken Turns via a Generalized Style-Aware Full-Duplex Framework4 Aug 2026
- 2026M3-DuplexBench: A Multi-Turn, Multilingual, Multidomain Benchmark for Full-Duplex Spoken Dialogue Models31 Jul 2026
- 2026Cocktail-Talker: Multi-Speaker Dialog Modeling in Noisy Social Environments with Turn Action GRPO30 Jul 2026
- 2026DuplexGen: Adaptive Synthesis of Human-AI Turn-Taking Dialogues28 Jul 2026
- 2026Low-Latency Turn-Taking via Context-Aware Preface Generation in a Real-World Dialogue Robot25 Jul 2026
- 2026Noisy-Channel Minimum Bayes Risk Decoding6 Jul 2026
- 2026DuplexChat: Constructing Speaker-Separated Full-Duplex Dialogue Speech at Scale for Spoken Dialogue Language Modeling6 Jul 2026
- 2026FacePlex: Toward Natural Full-Duplex Conversational Avatars29 Jun 2026
- 2026Overcoming State Inertia in Full-Duplex Spoken Language Models via Activation Steering9 Jun 2026
- 2026Multi-Faceted Interactivity Alignment in Full-Duplex Speech Models9 Jun 2026
- 2026IRAF: Interference-Resilient Adaptive Fusion for Noise-Robust End-to-End Full-Duplex Spoken Dialogue Systems4 Jun 2026
- 2026DyaPlex: Full-Duplex Speech-Motion Model for Dyadic Interaction2 Jun 2026
- 2026PolySpeech-100: A Large-Scale Benchmark for Speech Understanding Across 100+ Languages and Dialects31 May 2026
- 2026VideoFDB: Evaluating Full-Duplex Vision-Speech Capabilities in Conversational Agents28 May 2026
- 2026Instruct-FD: Can Your Full-Duplex Speech System Follow Turn-Taking Instructions?15 May 2026
- 2026Micro Language Models Enable Instant Responses21 Apr 2026
- 2026MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Models14 Apr 2026
- 2026ASPIRin: Action Space Projection for Interactivity-Optimized Reinforcement Learning in Full-Duplex Speech Language Models11 Apr 2026
- 2026Raon-Speech Technical Report8 Apr 2026
- 2026RelayS2S: A Dual-Path Speculative Generation for Real-Time Dialogue24 Mar 2026
50 newest of 57 All 57 (JSON)
- 2026VoiceChat-TTS: A Low-Latency Continuous Speech Synthesis Model for Interactive Agentscited by 3
- 2026MM-tau-p2: Persona-Adaptive Prompting for Robust Multi-Modal Agent Evaluation in Dual-Control Settingscited by 2
- 2026DuplexSpeechBench-IFEval: Evaluating Implicit Instruction Following in Full-Duplex Voice Agentscited by 1
- 2026A frontend-backend architecture for tool calls in full-duplex speech modelscited by 1
- 2026Agentic-DuplexGen: Decoupling Content, Timing, and Acoustics for Synthetic Dialogue Speechcited by 0
- 2026IndicFDB: Benchmarking Full-Duplex Voice Agents across Indian Languagescited by 0
- 2026IndicFDB: Benchmarking Full-Duplex Voice Agents across Indian Languages25 Sep 2026
- 2026A frontend-backend architecture for tool calls in full-duplex speech models16 Sep 2026
- 2026DuplexSpeechBench-IFEval: Evaluating Implicit Instruction Following in Full-Duplex Voice Agents3 Sep 2026
- 2026Agentic-DuplexGen: Decoupling Content, Timing, and Acoustics for Synthetic Dialogue Speech17 Aug 2026
- 2026VoiceChat-TTS: A Low-Latency Continuous Speech Synthesis Model for Interactive Agents13 Aug 2026
- 2026MM-tau-p2: Persona-Adaptive Prompting for Robust Multi-Modal Agent Evaluation in Dual-Control Settings10 Mar 2026
- 2026MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Modelscited by 11
- 2026Multi-Faceted Interactivity Alignment in Full-Duplex Speech Modelscited by 9
- 2026DuplexCascade: Full-Duplex Speech-to-Speech Dialogue with VAD-Free Cascaded ASR-LLM-TTS Pipeline and Micro-Turn Optimizationcited by 7
- 2026Raon-Speech Technical Reportcited by 4
- 2026Instruct-FD: Can Your Full-Duplex Speech System Follow Turn-Taking Instructions?cited by 4
- 2026ASPIRin: Action Space Projection for Interactivity-Optimized Reinforcement Learning in Full-Duplex Speech Language Modelscited by 3
- 2026DuplexChat: Constructing Speaker-Separated Full-Duplex Dialogue Speech at Scale for Spoken Dialogue Language Modelingcited by 3
- 2026NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilitiescited by 3
- 2026Overcoming State Inertia in Full-Duplex Spoken Language Models via Activation Steeringcited by 2
- 2026Cocktail-Talker: Multi-Speaker Dialog Modeling in Noisy Social Environments with Turn Action GRPOcited by 2
- 2026M3-DuplexBench: A Multi-Turn, Multilingual, Multidomain Benchmark for Full-Duplex Spoken Dialogue Modelscited by 2
- 2026MP-Bench: Evaluating Voice Agents as a Multiparty Conversation Participantcited by 2
12 most cited of 47 All 47 (JSON)
- 2026EgoVoice: Proactive Spoken Assistance from Egocentric Multimodal Streams8 Oct 2026
- 2026Prompts versus Rules: Auditing and Controlling Speech Naturalness Behaviors in Voice User Simulators7 Oct 2026
- 2026DuplexAct-Bench: Broadening Full-Duplex Speech Evaluation toward Proactive Interaction across Diverse Behavioral Requirements30 Sep 2026
- 2026EvolvingAvatar: Interactive 3D Head Generation That Adapts as Conversations Unfold28 Sep 2026
- 2026DuplexCadence: Exact State and Execution from a Speech Model's Declared Timelines28 Sep 2026
- 2026Context Spanning: A Communication Framework for Full-Duplex Speech Models and External LLM Backends27 Sep 2026
- 2026Controlling Backchannels in Streamable Full-duplex Models24 Sep 2026
- 2026Neither Silence nor Overlap Is Failure: Intent-Conditioned Evaluation of Turn-Taking in Full-Duplex Spoken Dialogue Models23 Sep 2026
- 2026Distributional Metrics for Evaluating Spoken Conversational Systems21 Sep 2026
- 2026NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilities18 Sep 2026
- 2026When Should a Failing Robot Ask? Initiating Corrective Human-Robot Dialogue from Audited Sensor Evidence18 Sep 2026
- 2026Full-Duplex Speech Models Take the Floor When Asked, Not When Needed17 Sep 2026
12 newest of 47 All 47 (JSON)