# KAME: Tandem Architecture for Enhancing Knowledge in Real-Time Speech-to-Speech Conversational AI

arXiv: https://arxiv.org/abs/2510.02327 · HTML: https://mlsanity.com/paper/2510.02327/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 19 references that are in the map.
- First posted on arXiv 26 Sep 2025.

## Builds on (most cited first, 16 of 16)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Qwen2.5-Omni Technical Report](https://mlsanity.com/paper/2503.20215/) (2025)
- [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024)
- [SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities](https://mlsanity.com/paper/2305.11000/) (2023)
- [Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming](https://mlsanity.com/paper/2408.16725/) (2024)
- [LLaMA-Omni: Seamless Speech Interaction with Large Language Models](https://mlsanity.com/paper/2409.06666/) (2024)
- [Generative Spoken Dialogue Language Modeling](https://mlsanity.com/paper/2203.16502/) (2022)
- [Deliberation Model Based Two-Pass End-to-End Speech Recognition](https://mlsanity.com/paper/2003.07962/) (2020)
- [CrisperWhisper: Accurate Timestamps on Verbatim Speech Transcriptions](https://mlsanity.com/paper/2408.16589/) (2024)
- [Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling](https://mlsanity.com/paper/2509.08753/) (2025)
- [Minions: Cost-efficient Collaboration Between On-device and Cloud Language Models](https://mlsanity.com/paper/2502.15964/) (2025)
- [HSSBench: Benchmarking Humanities and Social Sciences Ability for Multimodal Large Language Models](https://mlsanity.com/paper/2506.03922/) (2025)
- [Streaming Endpointer for Spoken Dialogue using Neural Audio Codecs and Label-Delayed Training](https://mlsanity.com/paper/2506.07081/) (2025)

## Built on by (most cited first, 7 of 7)

- [MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Models](https://mlsanity.com/paper/2604.12928/) (2026)
- [Thinking While Speaking: Inference-Time Knowledge Transfer for Responsive and Intelligent Conversational Voice Agents](https://mlsanity.com/paper/2511.07397/) (2025)
- [A frontend-backend architecture for tool calls in full-duplex speech models](https://mlsanity.com/paper/2609.19334/) (2026)
- [SALMONN-duo: Adaptive Dual-System Coordination for Full-Duplex Voice Agents](https://mlsanity.com/paper/2609.34247/) (2026)
- [Towards Expert-level Medical AI for Real-time Video Consultations](https://mlsanity.com/paper/2608.09861/) (2026)
- [Learning Natural Conversational Behavior in Tandem Speech-to-Speech Models with Randomized Guidance](https://mlsanity.com/paper/2609.30773/) (2026)
- [Context Spanning: A Communication Framework for Full-Duplex Speech Models and External LLM Backends](https://mlsanity.com/paper/2609.33443/) (2026)

## Abstract (arXiv)

Real-time speech-to-speech (S2S) models excel at generating natural, low-latency conversational responses but often lack deep knowledge and semantic understanding. Conversely, cascaded systems combining automatic speech recognition, a text-based Large Language Model (LLM), and text-to-speech synthesis offer superior knowledge representation at the cost of high latency, which disrupts the flow of natural interaction. This paper introduces a novel hybrid architecture that bridges the gap between these two paradigms. Our framework processes user speech through an S2S transformer for immediate responsiveness while concurrently relaying the query to a powerful back-end LLM. The LLM's text-based response is then injected in real time to guide the S2S model's speech generation, effectively infusing its output with rich knowledge without the full latency penalty of a cascaded system. We evaluated our method using a speech-synthesized variant of the MT-Bench benchmark that consists of multi-turn question-answering sessions. The results demonstrate that our system substantially outperforms a baseline S2S model in response correctness, approaching that of a cascaded system, while maintaining a latency on par with the baseline.