# Jarvis: A Proactive Speech Agent for Multi-Party Conversations

arXiv: https://arxiv.org/abs/2610.07506 · HTML: https://mlsanity.com/paper/2610.07506/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 88 references that are in the map.
- First posted on arXiv 5 Oct 2026.

## Builds on (most cited first, 18 of 18)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024)
- [Teaching language models to support answers with verified quotes](https://mlsanity.com/paper/2203.11147/) (2022)
- [Qwen3-TTS Technical Report](https://mlsanity.com/paper/2601.15621/) (2026)
- [Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling](https://mlsanity.com/paper/2509.08753/) (2025)
- [Stateful Conformer with Cache-based Inference for Streaming Automatic Speech Recognition](https://mlsanity.com/paper/2312.17279/) (2023)
- [ProAgentBench: Evaluating LLM Agents for Proactive Assistance with Real-World Data](https://mlsanity.com/paper/2602.04482/) (2026)
- [Multi-User Chat Assistant (MUCA): a Framework Using LLMs to Facilitate Group Conversations](https://mlsanity.com/paper/2401.04883/) (2024)
- [Beyond Reactivity: Measuring Proactive Problem Solving in LLM Agents](https://mlsanity.com/paper/2510.19771/) (2025)
- [From Reactive to Proactive: Assessing the Proactivity of Voice Agents via ProVoice-Bench](https://mlsanity.com/paper/2604.15037/) (2026)
- [Bringing Everyone to the Table: An Experimental Study of LLM-Facilitated Group Decision Making](https://mlsanity.com/paper/2508.08242/) (2025)
- [ProMediate: A Socio-cognitive framework for evaluating proactive agents in multi-party negotiation](https://mlsanity.com/paper/2510.25224/) (2025)
- [DiscussLLM: Teaching Large Language Models When to Speak](https://mlsanity.com/paper/2508.18167/) (2025)
- [Adaptive Turn-Taking for Real-time Multi-Party Voice Agents](https://mlsanity.com/paper/2606.13544/) (2026)
- [MeetBench-XL: Calibrated Multi-Dimensional Evaluation and Learned Dual-Policy Agents for Real-Time Meetings](https://mlsanity.com/paper/2602.03285/) (2026)
- [ProACT: Towards Breakdown-Aware Proactive Agent in Multi-User Collaboration](https://mlsanity.com/paper/2607.03730/) (2026)

## Abstract (arXiv)

Speech agents are reactive and dyadic: they speak when spoken to, and to one person at a time. We ask what it takes for a speech agent to instead take part in a conversation among several people and speak up only when it can help. We introduce Jarvis, a real-time proactive speech agent that audibly participates in multi-party human conversations. Grounded in a document shared beforehand, Jarvis follows the discussion and intervenes when the group misses or misstates a fact and does not correct itself within a few turns. We make three contributions: a problem setting based on epistemic breakdowns that makes proactive intervention measurable, realized as CHI-180-proactive, a synthetic multi-party dataset seeded with known gaps, errors, and self-corrections; a proactive backbone that harnesses a small, open-weight model with deterministic checks and grounds every claim in a source sentence; and interaction techniques for taking the floor in live speech and showing the cited evidence on screen. On CHI-180-proactive, Jarvis is correct on most events it addresses and stays silent 97% of the time when the group resolves an issue itself. A live study with 23 participants confirms these trends with real-time interventions.