# CrossTalk: Intelligent Substrates for Language-Oriented Interaction in Video-Based Communication and Collaboration

arXiv: https://arxiv.org/abs/2308.03311 · HTML: https://mlsanity.com/paper/2308.03311/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 74 references that are in the map.
- First posted on arXiv 7 Aug 2023.

## Builds on (most cited first, 3 of 3)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Meeting Effectiveness and Inclusiveness in Remote Collaboration](https://mlsanity.com/paper/2102.09803/) (2021)
- [MeetCues: Supporting Online Meetings Experience](https://mlsanity.com/paper/2010.06259/) (2020)

## Built on by (most cited first, 6 of 6)

- [Augmented Physics: Creating Interactive and Embedded Physics Simulations from Static Textbook Diagrams](https://mlsanity.com/paper/2405.18614/) (2024)
- [DrawTalking: Building Interactive Worlds by Sketching and Speaking](https://mlsanity.com/paper/2401.05631/) (2024)
- [Orca: Browsing at Scale Through User-Driven and AI-Facilitated Orchestration Across Malleable Webpages](https://mlsanity.com/paper/2505.22831/) (2025)
- [MeetMap: Real-Time Collaborative Dialogue Mapping with LLMs in Online Meetings](https://mlsanity.com/paper/2502.01564/) (2025)
- [MapStory: Prototyping Editable Map Animations with LLM Agents](https://mlsanity.com/paper/2505.21966/) (2025)
- [Spatula: Exploring On-Demand In-Situ Interfaces and Interaction for Attribute Control](https://mlsanity.com/paper/2607.10405/) (2026)

## Abstract (arXiv)

Despite the advances and ubiquity of digital communication media such as videoconferencing and virtual reality, they remain oblivious to the rich intentions expressed by users. Beyond transmitting audio, videos, and messages, we envision digital communication media as proactive facilitators that can provide unobtrusive assistance to enhance communication and collaboration. Informed by the results of a formative study, we propose three key design concepts to explore the systematic integration of intelligence into communication and collaboration, including the panel substrate, language-based intent recognition, and lightweight interaction techniques. We developed CrossTalk, a videoconferencing system that instantiates these concepts, which was found to enable a more fluid and flexible communication and collaboration experience.