Human-1 by Josh Talks: A Full-Duplex Conversational Modeling Framework in Hindi using Real-World Conversations
Not on the map. Why?
Abstract (arXiv)
Full-duplex spoken dialogue systems can model natural conversational behaviours such as interruptions, overlaps, and backchannels, yet such systems remain largely unexplored for Indian languages. We present the first open, reproducible full-duplex spoken dialogue system for Hindi by adapting Moshi, a state-of-the-art duplex speech architecture, using a custom Hindi tokeniser and training on 26,000 hours of real spontaneous conversations collected from 14,695 speakers with separate speaker channels, enabling direct learning of turn-taking and overlap patterns from natural interactions. To support Hindi text generation, we replace the original English tokeniser and reinitialise text-vocabulary-dependent parameters while retaining the pre-trained audio components. We propose a two-stage training recipe -- large-scale pre-training followed by fine-tuning on 1,000 hours of conversational data. Evaluation through the prompted dialogue continuation paradigm with both automatic metrics and human judgments demonstrates that the resulting model generates natural and meaningful full-duplex conversational behaviour in Hindi. This work serves as a first step toward real-time duplex spoken dialogue systems for Hindi and other Indian languages.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 17 of its 24 references that are in the map. First posted on arXiv 25 Apr 2026.
Builds on
17 of 24 references.
References · 17
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
- 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
- 2022High Fidelity Neural Audio Compressioncited by 673
- 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
- 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
- 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
- 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
- 2023WhisperX: Time-Accurate Speech Transcription of Long-Form Audiocited by 320
- 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
- 2022Generative Spoken Dialogue Language Modelingcited by 125
- 2025Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilitiescited by 94
5 more
- 2024Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agentscited by 69
- 2024Language Model Can Listen While Speakingcited by 62
- 2024OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversationcited by 57
- 2024A Full-duplex Speech Dialogue Scheme Based On Large Language Modelscited by 40
- 2024SALMONN-omni: A Codec-free LLM for Full-duplex Speech Understanding and Generationcited by 33
Built on by
Not cited yet.