mlsanityData to 7 Oct 2026

Make a Video Call with LLM: A Measurement Campaign over Six Mainstream Apps

arXiv 2510.00481 ↗First posted 1 Oct 2025cs.NI cs.AI cs.HCcited by 0

Not on the map. Why?

Abstract (arXiv)

In 2025, Large Language Model (LLM) services have launched a new feature -- AI video chat -- allowing users to interact with AI agents via real-time video communication (RTC), just like chatting with real people. Despite its significance, no systematic study has characterized the performance of existing AI video chat systems. To address this gap, this paper proposes a comprehensive benchmark across four dimensions: quality, latency, internal mechanisms, and system overhead. Using custom testbeds, we further evaluate six mainstream AI video chatbots with this benchmark. We also build an online platform for user study. The measurement leads to interesting findings that could be beneficial to the future optimizations. For example, the network latency of AI video chat matters not as much as human video chat. The capabilities of AI agents matters most in the user experience. Our benchmarking results also open up several research questions for future optimizations of AI video chatbots. Availability: https://callarena.net/ for the online evaluation platform and our open-sourced dataset and testbed.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 12 of its 49 references that are in the map. First posted on arXiv 1 Oct 2025.

Builds on

12 of 49 references.

References · 12

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  4. 2024GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbotcited by 218
  5. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 200
  6. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  7. 2024StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understandingcited by 135
  8. 2024A Survey of Useful LLM Evaluationcited by 28
  9. 2023Beyond Static Datasets: A Deep Interaction Approach to LLM Evaluationcited by 24
  10. 2021Measuring the Performance and Network Utilization of Popular Video Conferencing Applicationscited by 10
  11. 2024KV Prediction for Improved Time to First Tokencited by 9
  12. 2023Toward One-Second Latency: Evolution of Live Media Streamingcited by 3

Built on by

Not cited yet.