# QuAVF: Quality-aware Audio-Visual Fusion for Ego4D Talking to Me Challenge

arXiv: https://arxiv.org/abs/2306.17404 · HTML: https://mlsanity.com/paper/2306.17404/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 6 references that are in the map.
- First posted on arXiv 30 Jun 2023.

## Builds on (most cited first, 6 of 6)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction](https://mlsanity.com/paper/2201.02184/) (2022)
- [Egocentric Video Task Translation @ Ego4D Challenge 2022](https://mlsanity.com/paper/2302.01891/) (2023)

## Built on by (most cited first, 6 of 6)

- [The Audio-Visual Conversational Graph: From an Egocentric-Exocentric Perspective](https://mlsanity.com/paper/2312.12870/) (2023)
- [PCIE_Interaction Solution for Ego4D Social Interaction Challenge](https://mlsanity.com/paper/2505.24404/) (2025)
- [GateFusion: Hierarchical Gated Cross-Modal Fusion for Active Speaker Detection](https://mlsanity.com/paper/2512.15707/) (2025)
- [EgoAdapt: Enhancing Robustness in Egocentric Interactive Speaker Detection Under Missing Modalities](https://mlsanity.com/paper/2603.18082/) (2026)
- [Selective Attention System (SAS): Device-Addressed Speech Detection for Real-Time On-Device Voice AI](https://mlsanity.com/paper/2604.08412/) (2026)
- [Talking to Me or Someone Else? Rethinking Talk-to-Me Detection in Egocentric Videos](https://mlsanity.com/paper/2609.14118/) (2026)

## Abstract (arXiv)

This technical report describes our QuAVF@NTU-NVIDIA submission to the Ego4D Talking to Me (TTM) Challenge 2023. Based on the observation from the TTM task and the provided dataset, we propose to use two separate models to process the input videos and audio. By doing so, we can utilize all the labeled training data, including those without bounding box labels. Furthermore, we leverage the face quality score from a facial landmark prediction model for filtering noisy face input data. The face quality score is also employed in our proposed quality-aware fusion for integrating the results from two branches. With the simple architecture design, our model achieves 67.4% mean average precision (mAP) on the test set, which ranks first on the leaderboard and outperforms the baseline method by a large margin. Code is available at: https://github.com/hsi-che-lin/Ego4D-QuAVF-TTM-CVPR23