# Modeling Real-Time Interactive Conversations as Timed Diarized Transcripts

arXiv: https://arxiv.org/abs/2405.13203 · HTML: https://mlsanity.com/paper/2405.13203/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 35 references that are in the map.
- First posted on arXiv 21 May 2024.

## Builds on (most cited first, 14 of 14)

- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019)
- [A decoder-only foundation model for time-series forecasting](https://mlsanity.com/paper/2310.10688/) (2023)
- [Recipes for building an open-domain chatbot](https://mlsanity.com/paper/2004.13637/) (2020)
- [Towards a Human-like Open-Domain Chatbot](https://mlsanity.com/paper/2001.09977/) (2020)
- [WhisperX: Time-Accurate Speech Transcription of Long-Form Audio](https://mlsanity.com/paper/2303.00747/) (2023)
- [Internet-Augmented Dialogue Generation](https://mlsanity.com/paper/2107.07566/) (2021)
- [Streaming Simultaneous Speech Translation with Augmented Memory Transformer](https://mlsanity.com/paper/2011.00033/) (2020)
- [Incremental Text-to-Speech Synthesis with Prefix-to-Prefix Framework](https://mlsanity.com/paper/1911.02750/) (2019)
- [Generative Ghosts: Anticipating Benefits and Risks of AI Afterlives](https://mlsanity.com/paper/2402.01662/) (2024)
- [The timing bottleneck: Why timing and overlap are mission-critical for conversational user interfaces, speech recognition and dialogue systems](https://mlsanity.com/paper/2307.15493/) (2023)

## Built on by (most cited first, 1 of 1)

- [Salmon: A Suite for Acoustic Language Model Evaluation](https://mlsanity.com/paper/2409.07437/) (2024)

## Abstract (arXiv)

Chatbots built upon language models have exploded in popularity, but they have largely been limited to synchronous, turn-by-turn dialogues. In this paper we present a simple yet general method to simulate real-time interactive conversations using pretrained text-only language models, by modeling timed diarized transcripts and decoding them with causal rejection sampling. We demonstrate the promise of this method with two case studies: instant messenger dialogues and spoken conversations, which require generation at about 30 tok/s and 20 tok/s respectively to maintain real-time interactivity. These capabilities can be added into language models using relatively little data and run on commodity hardware.