# i-LAVA: Insights on Low Latency Voice-2-Voice Architecture for Agents

arXiv: https://arxiv.org/abs/2509.20971 · HTML: https://mlsanity.com/paper/2509.20971/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 13 references that are in the map.
- First posted on arXiv 25 Sep 2025.

## Builds on (most cited first, 7 of 7)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Evaluating the Efficacy of Open-Source LLMs in Enterprise-Specific RAG Systems: A Comparative Study of Performance and Scalability](https://mlsanity.com/paper/2406.11424/) (2024)
- [Keyword Augmented Retrieval: Novel framework for Information Retrieval integrated with speech interface](https://mlsanity.com/paper/2310.04205/) (2023)
- [Introducing a new hyper-parameter for RAG: Context Window Utilization](https://mlsanity.com/paper/2407.19794/) (2024)
- [VLMs-in-the-Wild: Bridging the Gap Between Academic Benchmarks and Enterprise Reality](https://mlsanity.com/paper/2509.06994/) (2025)
- [E-ARMOR: Edge case Assessment and Review of Multilingual Optical Character Recognition](https://mlsanity.com/paper/2509.03615/) (2025)
- [M-PACE: Mother Child Framework for Multimodal Compliance](https://mlsanity.com/paper/2509.15241/) (2025)

## Built on by (most cited first, 2 of 2)

- [Benchmarking LLM Judges for Voice-Agent Evaluation: Reliability, Calibration, and Human Oversight](https://mlsanity.com/paper/2608.24314/) (2026)
- [When Fine-Tuning Fails and when it Generalises: Role of Data Diversity and Mixed Training in LLM-based TTS](https://mlsanity.com/paper/2603.10904/) (2026)

## Abstract (arXiv)

We experiment with a low-latency, end-to-end voice-to-voice communication model to optimize it for real-time conversational applications. By analyzing components essential to voice to voice (V-2-V) system viz. automatic speech recognition (ASR), text-to-speech (TTS), and dialog management, our work analyzes how to reduce processing time while maintaining high-quality interactions to identify the levers for optimizing V-2-V system. Our work identifies that TTS component which generates life-like voice, full of emotions including natural pauses and exclamations has highest impact on Real time factor (RTF). The experimented V-2-V architecture utilizes CSM1b has the capability to understand tone as well as context of conversation by ingesting both audio and text of prior exchanges to generate contextually accurate speech. We explored optimization of Residual Vector Quantization (RVQ) iterations by the TTS decoder which come at a cost of decrease in the quality of voice generated. Our experimental evaluations also demonstrate that for V-2-V implementations based on CSM most important optimizations can be brought by reducing the number of RVQ Iterations along with the codebooks used in Mimi.