# Thinking While Speaking: Inference-Time Knowledge Transfer for Responsive and Intelligent Conversational Voice Agents arXiv: https://arxiv.org/abs/2511.07397 ยท HTML: https://mlsanity.com/paper/2511.07397/ ## Key facts - Cited by 2 papers in the map (data through 14 Oct 2026). - Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning. - Builds on 31 of its 76 references that are in the map. - First posted on arXiv 10 Nov 2025. ## Placed in - LLM reasoning / [Audio-language model reasoning](https://mlsanity.com/lr/audio-language-model-reasoning/) - LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/) ## Builds on (most cited first, 30 of 31) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022) - [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024) - [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020) - [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025) - [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025) - [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021) - [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023) - [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023) - [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018) - [Qwen2.5-Omni Technical Report](https://mlsanity.com/paper/2503.20215/) (2025) - [An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning](https://mlsanity.com/paper/2308.08747/) (2023) - [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024) - [Scaling Synthetic Data Creation with 1,000,000,000 Personas](https://mlsanity.com/paper/2406.20094/) (2024) - [Measuring short-form factuality in large language models](https://mlsanity.com/paper/2411.04368/) (2024) - [SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model](https://mlsanity.com/paper/2502.02737/) (2025) - [Online Speculative Decoding](https://mlsanity.com/paper/2310.07177/) (2023) - [Understanding the Performance and Estimating the Cost of LLM Fine-Tuning](https://mlsanity.com/paper/2408.04693/) (2024) - [MELTing point: Mobile Evaluation of Language Transformers](https://mlsanity.com/paper/2403.12844/) (2024) - [Agents Thinking Fast and Slow: A Talker-Reasoner Architecture](https://mlsanity.com/paper/2410.08328/) (2024) - [Mini-Omni-Reasoner: Token-Level Thinking-in-Speaking in Large Speech Models](https://mlsanity.com/paper/2508.15827/) (2025) - [WavRAG: Audio-Integrated Retrieval Augmented Generation for Spoken Dialogue Models](https://mlsanity.com/paper/2502.14727/) (2025) - [LUCY: Linguistic Understanding and Control Yielding Early Stage of Her](https://mlsanity.com/paper/2501.16327/) (2025) - [Rethinking Learning Rate Tuning in the Era of Large Language Models](https://mlsanity.com/paper/2309.08859/) (2023) - [Lessons From an App Update at Replika AI: Identity Discontinuity in Human-AI Relationships](https://mlsanity.com/paper/2412.14190/) (2024) - [Stream RAG: Instant and Accurate Spoken Dialogue Systems with Streaming Tool Usage](https://mlsanity.com/paper/2510.02044/) (2025) - [SplitReason: Learning To Offload Reasoning](https://mlsanity.com/paper/2504.16379/) (2025) - [PredGen: Accelerated Inference of Large Language Models through Input-Time Speculation for Real-Time Speech Interaction](https://mlsanity.com/paper/2506.15556/) (2025) - [Voice Evaluation of Reasoning Ability: Diagnosing the Modality-Induced Performance Gap](https://mlsanity.com/paper/2509.26542/) (2025) - [KAME: Tandem Architecture for Enhancing Knowledge in Real-Time Speech-to-Speech Conversational AI](https://mlsanity.com/paper/2510.02327/) (2025) ## Built on by (most cited first, 2 of 2) - [PACE: Perceived-Latency-Aware Cascading Service Routing and Filler Control for QoE-Efficient Retrieval-Augmented Dialogue Serving](https://mlsanity.com/paper/2609.10372/) (2026) - [Learning Natural Conversational Behavior in Tandem Speech-to-Speech Models with Randomized Guidance](https://mlsanity.com/paper/2609.30773/) (2026) ## Abstract (arXiv) Voice agents face a fundamental tension: the reasoning, retrieval, and tool use that make foundation models capable are iterative and slow, while conversational interaction demands responses on a millisecond timescale. Smaller, real-time models meet the latency bar but cannot match foundation models on complex tasks, leaving current voice agents to trade away either responsiveness or capability. We introduce conversational infill, where a small talker model both immediately generates contextually grounded responses to hide the latency of an external reasoner model and fluently integrates streamed reasoner knowledge into its responses during inference. We curate a 290,571-example synthetic dataset spanning six domains and demonstrate that this task is learnable across seven widely used small language models ranging from 135M to 1.7B parameters. Our system implementation, ConvFill, sustains millisecond-level time-to-first-response while closing the accuracy gap to within 6.3% of the corresponding frontier reasoner performance. In a live user study (n=18) with talker deployments running on an Apple M2 SoC, participants rank ConvFill on par with frontier models overall, prefer it for retrieval-heavy tasks, and rate it significantly more responsive. These results show that conversational infill unlocks a new point on the latency-capability Pareto frontier, offering a practical path toward voice agents that are both responsive and highly capable. Code, models, and datasets are available at https://github.com/vysri/conversational-infill.