F-Actor: Controllable Conversational Behaviour in Full-Duplex Models
Not on the map. Why?
Abstract (arXiv)
Spoken conversational systems require more than accurate speech generation to have human-like conversations: to feel natural and engaging, they must produce conversational behaviour that adapts dynamically to the context. Current spoken conversational systems, however, rarely allow such customization, limiting their naturalness and usability. In this work, we present the first open, instruction-following full-duplex conversational speech model that can be trained efficiently under typical academic resource constraints. By keeping the audio encoder frozen and finetuning only the language model, our model requires just 2,000 hours of data, without relying on large-scale pretraining or multi-stage optimization. The model can follow explicit instructions to control speaker voice, conversation topic, conversational behaviour (e.g., backchanneling and interruptions), and dialogue initiation. We propose a single-stage training protocol and systematically analyze design choices. Both the model and training code is released to enable reproducible research on controllable full-duplex speech systems.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 28 of its 60 references that are in the map. First posted on arXiv 16 Jan 2026.
Builds on
28 of 60 references.
References · 28
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
- 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
- 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
- 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 425
- 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 381
- 2021Generative Spoken Language Modeling from Raw Audiocited by 336
- 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 323
- 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 293
- 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 191
- 2024Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLMcited by 135
- 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
16 more
- 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
- 2023Powerset multi-class cross entropy loss for neural speaker diarizationcited by 109
- 2025MinMo: A Multimodal Large Language Model for Seamless Voice Interactioncited by 79
- 2024Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agentscited by 69
- 2025Full-Duplex-Bench v1.5: Evaluating Overlap Handling for Full-Duplex Speech Modelscited by 53
- 2021ECAPA-TDNN Embeddings for Speaker Diarizationcited by 47
- 2025SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Modelcited by 32
- 2024AnthroScore: A Computational Linguistic Measure of Anthropomorphismcited by 27
- 2025NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pair Predictioncited by 19
- 2025FireRedChat: A Pluggable, Full-Duplex Voice Interaction System with Cascaded and Semi-Cascaded Implementationscited by 17
- 2025FlexDuo: A Pluggable System for Enabling Full-Duplex Capabilities in Speech Dialogue Systemscited by 15
- 2025NanoCodec: Towards High-Quality Ultra Fast Speech LLM Inferencecited by 14
- 2017Yeah, Right, Uh-Huh: A Deep Learning Backchannel Predictorcited by 12
- 2025From Turn-Taking to Synchronous Dialogue: A Survey of Full-Duplex Spoken Language Modelscited by 9
- 2022Alexa as an Active Listener: How Backchanneling Can Elicit Self-Disclosure and Promote User Experiencecited by 6
- 2025Segmentation-Variant Codebooks for Preservation of Paralinguistic and Prosodic Informationcited by 4
Built on by
Citing papers per year: 2026: 2.
2 papers.