mlsanityData to 7 Oct 2026

PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models

arXiv 2602.06053 ↗First posted 14 Jan 2026cs.CLcited by 57

Not on the map. Why?

Abstract (arXiv)

Recent advances in duplex speech models have enabled natural, low-latency speech-to-speech interactions. However, existing models are restricted to a fixed role and voice, limiting their ability to support structured, role-driven real-world applications and personalized interactions. In this work, we introduce PersonaPlex, a duplex conversational speech model that incorporates hybrid system prompts, combining role conditioning with text prompts and voice cloning with speech samples. PersonaPlex is trained on a large-scale synthetic dataset of paired prompts and user-agent conversations, generated with open-source large language models (LLM) and text-to-speech (TTS) models. To evaluate role conditioning in real-world settings, we extend the Full-Duplex-Bench benchmark beyond a single assistant role to multi-role customer service scenarios. Experiments show that PersonaPlex achieves strong role-conditioned behavior, voice-conditioned speech, and natural conversational responsiveness, surpassing state-of-the-art duplex speech models and hybrid large language model-based speech systems in role adherence, speaker similarity, latency, and naturalness.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Builds on 23 of its 33 references that are in the map. First posted on arXiv 14 Jan 2026.

Builds on

23 of 33 references.

References · 23

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  4. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  5. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  6. 2025Qwen2.5-Omni Technical Reportcited by 704
  7. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  8. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  9. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  10. 2024GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbotcited by 218
  11. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 204
  12. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 192
11 more
  1. 2024Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLMcited by 135
  2. 2024Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilitiescited by 104
  3. 2025Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilitiescited by 94
  4. 2023Libriheavy: a 50,000 hours ASR corpus with punctuation casing and contextcited by 84
  5. 2025LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesiscited by 75
  6. 2024Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agentscited by 69
  7. 2024OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversationcited by 57
  8. 2025SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Modelcited by 32
  9. 2025URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Modelscited by 29
  10. 2025Streaming Sequence-to-Sequence Learning with Delayed Streams Modelingcited by 24
  11. 2025VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Modelscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 57.

57 papers.

Sort:

By area:

  1. 2026MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Modelscited by 11
  2. 2026Multi-Faceted Interactivity Alignment in Full-Duplex Speech Modelscited by 9
  3. 2026DuplexCascade: Full-Duplex Speech-to-Speech Dialogue with VAD-Free Cascaded ASR-LLM-TTS Pipeline and Micro-Turn Optimizationcited by 7
  4. 2026Raon-Speech Technical Reportcited by 4
  5. 2026Instruct-FD: Can Your Full-Duplex Speech System Follow Turn-Taking Instructions?cited by 4
  6. 2026VoxServe: Streaming-Centric Serving System for Speech Language Modelscited by 3
  7. 2026ASPIRin: Action Space Projection for Interactivity-Optimized Reinforcement Learning in Full-Duplex Speech Language Modelscited by 3
  8. 2026DuplexChat: Constructing Speaker-Separated Full-Duplex Dialogue Speech at Scale for Spoken Dialogue Language Modelingcited by 3
  9. 2026VoiceChat-TTS: A Low-Latency Continuous Speech Synthesis Model for Interactive Agentscited by 3
  10. 2026NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilitiescited by 3
  11. 2026DualTurn: Learning Turn-Taking from Dual-Channel Generative Speech Pretrainingcited by 2
  12. 2026MM-tau-p2: Persona-Adaptive Prompting for Robust Multi-Modal Agent Evaluation in Dual-Control Settingscited by 2
38 more
  1. 2026Overcoming State Inertia in Full-Duplex Spoken Language Models via Activation Steeringcited by 2
  2. 2026Cocktail-Talker: Multi-Speaker Dialog Modeling in Noisy Social Environments with Turn Action GRPOcited by 2
  3. 2026M3-DuplexBench: A Multi-Turn, Multilingual, Multidomain Benchmark for Full-Duplex Spoken Dialogue Modelscited by 2
  4. 2026MP-Bench: Evaluating Voice Agents as a Multiparty Conversation Participantcited by 2
  5. 2026SpokenUS: A Spoken User Simulator for Task-Oriented Dialoguecited by 1
  6. 2026TiCo: Time-Controllable Spoken Dialogue Modelcited by 1
  7. 2026RelayS2S: A Dual-Path Speculative Generation for Real-Time Dialoguecited by 1
  8. 2026VideoFDB: Evaluating Full-Duplex Vision-Speech Capabilities in Conversational Agentscited by 1
  9. 2026DyaPlex: Full-Duplex Speech-Motion Model for Dyadic Interactioncited by 1
  10. 2026X2-Turn: Frame-Synchronous Dual-Head Modeling for Joint Streaming ASR and Turn State Predictioncited by 1
  11. 2026DuplexSpeechBench-IFEval: Evaluating Implicit Instruction Following in Full-Duplex Voice Agentscited by 1
  12. 2026Noise Adaptive Streaming Audio-Visual Speech Token Enhancement for Robust Full-Duplex Spoken Dialogue Modelscited by 1
  13. 2026DuplexDrama: A Synthesized Dialogue Dataset with Scenarios, Full-Duplex Behaviors, Expressive Speech, and Sound Eventscited by 1
  14. 2026Enabling Streaming User Transcription in Full-Duplex Speech-to-Speech Modelscited by 1
  15. 2026A frontend-backend architecture for tool calls in full-duplex speech modelscited by 1
  16. 2026Full-Duplex Speech Models Take the Floor When Asked, Not When Neededcited by 1
  17. 2026Sommelier: Scalable Open Multi-turn Audio Pre-processing for Full-duplex Speech Language Modelscited by 0
  18. 2026Micro Language Models Enable Instant Responsescited by 0
  19. 2026PolySpeech-100: A Large-Scale Benchmark for Speech Understanding Across 100+ Languages and Dialectscited by 0
  20. 2026IRAF: Interference-Resilient Adaptive Fusion for Noise-Robust End-to-End Full-Duplex Spoken Dialogue Systemscited by 0
  21. 2026FacePlex: Toward Natural Full-Duplex Conversational Avatarscited by 0
  22. 2026Noisy-Channel Minimum Bayes Risk Decodingcited by 0
  23. 2026Low-Latency Turn-Taking via Context-Aware Preface Generation in a Real-World Dialogue Robotcited by 0
  24. 2026DuplexGen: Adaptive Synthesis of Human-AI Turn-Taking Dialoguescited by 0
  25. 2026Agentic-DuplexGen: Decoupling Content, Timing, and Acoustics for Synthetic Dialogue Speechcited by 0
  26. 2026Does Listening Matter? Backchanneling and Nodding in AI Clonecited by 0
  27. 2026Enabling Proactive Spoken Turns via a Generalized Style-Aware Full-Duplex Frameworkcited by 0
  28. 2026KABURI-TTS: Phoneme-Keyed Activity-conditioned Bi-channel Utterance Rendering for Interactioncited by 0
  29. 2026DuplexJail: Safety Alignment Breaks Under Spoken Interruption in Full-Duplex Modelscited by 0
  30. 2026RetroThinker: Enabling Retrospective Thinking in Speech LLMscited by 0
  31. 2026SteerDuplex: Steerable Duplex Speech Dialogue Modelscited by 0
  32. 2026Continue, Adapt, or Yield: In-Turn Adaptation to Overlapping Speech in Full-Duplex Agentscited by 0
  33. 2026Causal Analysis and Mitigation of Spurious Onsets in Full-Duplex Speech LLMscited by 0
  34. 2026RoleBreak: Benchmarking Long-Horizon Role-Playing Robustness in Spoken Dialoguecited by 0
  35. 2026When Should a Failing Robot Ask? Initiating Corrective Human-Robot Dialogue from Audited Sensor Evidencecited by 0
  36. 2026Neither Silence nor Overlap Is Failure: Intent-Conditioned Evaluation of Turn-Taking in Full-Duplex Spoken Dialogue Modelscited by 0
  37. 2026Controlling Backchannels in Streamable Full-duplex Modelscited by 0
  38. 2026Distributional Metrics for Evaluating Spoken Conversational Systemscited by 0

50 most cited of 57 All 57 (JSON)