# The Audio-Visual Conversational Graph: From an Egocentric-Exocentric Perspective

arXiv: https://arxiv.org/abs/2312.12870 · HTML: https://mlsanity.com/paper/2312.12870/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 50 references that are in the map.
- First posted on arXiv 20 Dec 2023.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition](https://mlsanity.com/paper/1908.08498/) (2019)
- [Audiovisual SlowFast Networks for Video Recognition](https://mlsanity.com/paper/2001.08740/) (2020)
- [Future Person Localization in First-Person Videos](https://mlsanity.com/paper/1711.11217/) (2017)
- [VisualEchoes: Spatial Image Representation Learning through Echolocation](https://mlsanity.com/paper/2005.01616/) (2020)
- [In the Eye of Transformer: Global-Local Correlation for Egocentric Gaze Estimation](https://mlsanity.com/paper/2208.04464/) (2022)
- [4D Human Body Capture from Egocentric Video via 3D Scene Grounding](https://mlsanity.com/paper/2011.13341/) (2020)
- [With a Little Help from my Temporal Context: Multimodal Egocentric Action Recognition](https://mlsanity.com/paper/2111.01024/) (2021)
- [Egocentric Audio-Visual Object Localization](https://mlsanity.com/paper/2303.13471/) (2023)
- [Egocentric Deep Multi-Channel Audio-Visual Active Speaker Localization](https://mlsanity.com/paper/2201.01928/) (2022)
- [Learning State-Aware Visual Representations from Audible Interactions](https://mlsanity.com/paper/2209.13583/) (2022)
- [Egocentric Auditory Attention Localization in Conversations](https://mlsanity.com/paper/2303.16024/) (2023)
- [Generative Adversarial Network for Future Hand Segmentation from Egocentric Video](https://mlsanity.com/paper/2203.11305/) (2022)
- [Cross-Domain First Person Audio-Visual Action Recognition through Relative Norm Alignment](https://mlsanity.com/paper/2106.01689/) (2021)
- [Sonicverse: A Multisensory Simulation Platform for Embodied Household Agents that See and Hear](https://mlsanity.com/paper/2306.00923/) (2023)
- [Chat2Map: Efficient Scene Mapping from Multi-Ego Conversations](https://mlsanity.com/paper/2301.02184/) (2023)
- [QuAVF: Quality-aware Audio-Visual Fusion for Ego4D Talking to Me Challenge](https://mlsanity.com/paper/2306.17404/) (2023)
- [Conversation Group Detection With Spatio-Temporal Context](https://mlsanity.com/paper/2206.02559/) (2022)
- [Egocentric Video Task Translation @ Ego4D Challenge 2022](https://mlsanity.com/paper/2302.01891/) (2023)

## Built on by (most cited first, 14 of 14)

- [Challenges and Trends in Egocentric Vision: A Survey](https://mlsanity.com/paper/2503.15275/) (2025)
- [SocialGesture: Delving into Multi-person Gesture Understanding](https://mlsanity.com/paper/2504.02244/) (2025)
- [EGOILLUSION: Benchmarking Hallucinations in Egocentric Video Understanding](https://mlsanity.com/paper/2508.12687/) (2025)
- [Multi-speaker Attention Alignment for Multimodal Social Interaction](https://mlsanity.com/paper/2511.17952/) (2025)
- [Bridging Perspectives: A Survey on Cross-view Collaborative Intelligence with Egocentric-Exocentric Vision](https://mlsanity.com/paper/2506.06253/) (2025)
- [Egocentric and Exocentric Methods: A Short Survey](https://mlsanity.com/paper/2410.20621/) (2024)
- [Spherical World-Locking for Audio-Visual Localization in Egocentric Videos](https://mlsanity.com/paper/2408.05364/) (2024)
- [EgoAdapt: Adaptive Multisensory Distillation and Policy Learning for Efficient Egocentric Perception](https://mlsanity.com/paper/2506.21080/) (2025)
- [Interaction-aware Representation Modeling with Co-occurrence Consistency for Egocentric Hand-Object Parsing](https://mlsanity.com/paper/2602.20597/) (2026)
- [SocialDirector: Training-Free Social Interaction Control for Multi-Person Video Generation](https://mlsanity.com/paper/2605.10079/) (2026)
- [The Semantic Lifecycle in Embodied AI: Acquisition, Representation and Storage via Foundation Models](https://mlsanity.com/paper/2601.08876/) (2026)
- [Identification of Conversation Partners from Egocentric Video](https://mlsanity.com/paper/2406.08089/) (2024)
- [Talking to Me or Someone Else? Rethinking Talk-to-Me Detection in Egocentric Videos](https://mlsanity.com/paper/2609.14118/) (2026)
- [AI Smart Glasses for Wearable Intelligence: From Egocentric Sensing to Agentic Personalization](https://mlsanity.com/paper/2609.19793/) (2026)

## Abstract (arXiv)

In recent years, the thriving development of research related to egocentric videos has provided a unique perspective for the study of conversational interactions, where both visual and audio signals play a crucial role. While most prior work focus on learning about behaviors that directly involve the camera wearer, we introduce the Ego-Exocentric Conversational Graph Prediction problem, marking the first attempt to infer exocentric conversational interactions from egocentric videos. We propose a unified multi-modal framework -- Audio-Visual Conversational Attention (AV-CONV), for the joint prediction of conversation behaviors -- speaking and listening -- for both the camera wearer as well as all other social partners present in the egocentric video. Specifically, we adopt the self-attention mechanism to model the representations across-time, across-subjects, and across-modalities. To validate our method, we conduct experiments on a challenging egocentric video dataset that includes multi-speaker and multi-conversation scenarios. Our results demonstrate the superior performance of our method compared to a series of baselines. We also present detailed ablation studies to assess the contribution of each component in our model. Check our project page at https://vjwq.github.io/AV-CONV/.