# DMC3: Dual-Modal Counterfactual Contrastive Construction for Egocentric Video Question Answering

arXiv: https://arxiv.org/abs/2510.20285 · HTML: https://mlsanity.com/paper/2510.20285/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 60 references that are in the map.
- First posted on arXiv 23 Oct 2025.

## Builds on (most cited first, 29 of 29)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [Rescaling Egocentric Vision](https://mlsanity.com/paper/2006.13256/) (2020)
- [Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling](https://mlsanity.com/paper/2102.06183/) (2021)
- [VideoAgent: Long-form Video Understanding with Large Language Model as Agent](https://mlsanity.com/paper/2403.10517/) (2024)
- [Understanding Human Hands in Contact at Internet Scale](https://mlsanity.com/paper/2006.06669/) (2020)
- [EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought](https://mlsanity.com/paper/2305.15021/) (2023)
- [Egocentric Video-Language Pretraining](https://mlsanity.com/paper/2206.01670/) (2022)
- [Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?](https://mlsanity.com/paper/2303.18240/) (2023)
- [All in One: Exploring Unified Video-Language Pre-training](https://mlsanity.com/paper/2203.07303/) (2022)
- [EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbone](https://mlsanity.com/paper/2307.05463/) (2023)
- [EgoTaskQA: Understanding Human Tasks in Egocentric Videos](https://mlsanity.com/paper/2210.03929/) (2022)
- [EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World](https://mlsanity.com/paper/2403.16182/) (2024)
- [Video as Conditional Graph Hierarchy for Multi-Granular Question Answering](https://mlsanity.com/paper/2112.06197/) (2021)
- [Grounded Question-Answering in Long Egocentric Videos](https://mlsanity.com/paper/2312.06505/) (2023)
- [LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activities](https://mlsanity.com/paper/2007.15781/) (2020)
- [Grounded Multi-Hop VideoQA in Long-Form Egocentric Videos](https://mlsanity.com/paper/2408.14469/) (2024)
- [EgoTextVQA: Towards Egocentric Scene-Text Aware Video Question Answering](https://mlsanity.com/paper/2502.07411/) (2025)
- [Ego-Only: Egocentric Action Detection without Exocentric Transferring](https://mlsanity.com/paper/2301.01380/) (2023)
- [Helping Hands: An Object-Aware Ego-Centric Video Recognition Model](https://mlsanity.com/paper/2308.07918/) (2023)
- [EgoEnv: Human-centric environment representations from egocentric video](https://mlsanity.com/paper/2207.11365/) (2022)
- [ActionAtlas: A VideoQA Benchmark for Domain-specialized Action Recognition](https://mlsanity.com/paper/2410.05774/) (2024)
- [Hierarchical Conditional Relation Networks for Multimodal Video Question Answering](https://mlsanity.com/paper/2010.10019/) (2020)
- [A Feature-space Multimodal Data Augmentation Technique for Text-video Retrieval](https://mlsanity.com/paper/2208.02080/) (2022)
- [Do Vision-Language Transformers Exhibit Visual Commonsense? An Empirical Study of VCR](https://mlsanity.com/paper/2405.16934/) (2024)
- [Enhancing Vision-Language Pre-Training with Jointly Learned Questioner and Dense Captioner](https://mlsanity.com/paper/2305.11769/) (2023)

## Abstract (arXiv)

Egocentric Video Question Answering (Egocentric VideoQA) plays an important role in egocentric video understanding, which refers to answering questions based on first-person videos. Although existing methods have made progress through the paradigm of pre-training and fine-tuning, they ignore the unique challenges posed by the first-person perspective, such as understanding multiple events and recognizing hand-object interactions. To deal with these challenges, we propose a Dual-Modal Counterfactual Contrastive Construction (DMC3) framework, which contains an egocentric videoqa baseline, a counterfactual sample construction module and a counterfactual sample-involved contrastive optimization. Specifically, We first develop a counterfactual sample construction module to generate positive and negative samples for textual and visual modalities through event description paraphrasing and core interaction mining, respectively. Then, We feed these samples together with the original samples into the baseline. Finally, in the counterfactual sample-involved contrastive optimization module, we apply contrastive loss to minimize the distance between the original sample features and the positive sample features, while maximizing the distance from the negative samples. Experiments show that our method achieve 52.51% and 46.04% on the textitnormal and textitindirect splits of EgoTaskQA, and 13.2% on QAEGO4D, both reaching the state-of-the-art performance.