# HCQA @ Ego4D EgoSchema Challenge 2024

arXiv: https://arxiv.org/abs/2406.15771 · HTML: https://mlsanity.com/paper/2406.15771/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 26 references that are in the map.
- First posted on arXiv 22 Jun 2024.

## Builds on (most cited first, 17 of 17)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://mlsanity.com/paper/2304.14178/) (2023)
- [EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding](https://mlsanity.com/paper/2308.09126/) (2023)
- [VideoAgent: Long-form Video Understanding with Large Language Model as Agent](https://mlsanity.com/paper/2403.10517/) (2024)
- [Egocentric Video-Language Pretraining](https://mlsanity.com/paper/2206.01670/) (2022)
- [Learning Video Representations from Large Language Models](https://mlsanity.com/paper/2212.04501/) (2022)
- [Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers](https://mlsanity.com/paper/2212.10559/) (2022)
- [A Simple LLM Framework for Long-Range Video Question-Answering](https://mlsanity.com/paper/2312.17235/) (2023)
- [Video ReCap: Recursive Captioning of Hour-Long Videos](https://mlsanity.com/paper/2402.13250/) (2024)
- [EmoCaps: Emotion Capsule based Model for Conversational Emotion Recognition](https://mlsanity.com/paper/2203.13504/) (2022)
- [Zero-Shot Video Question Answering with Procedural Programs](https://mlsanity.com/paper/2312.00937/) (2023)
- [A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Frames](https://mlsanity.com/paper/2312.07395/) (2023)
- [Enhancing Emotional Generation Capability of Large Language Models via Emotional Chain-of-Thought](https://mlsanity.com/paper/2401.06836/) (2024)
- [UniSA: Unified Generative Framework for Sentiment Analysis](https://mlsanity.com/paper/2309.01339/) (2023)

## Built on by (most cited first, 9 of 9)

- [Spatial Understanding from Videos: Structured Prompts Meet Simulation Data](https://mlsanity.com/paper/2506.03642/) (2025)
- [Exo2Ego: Exocentric Knowledge Guided MLLM for Egocentric Video Understanding](https://mlsanity.com/paper/2503.09143/) (2025)
- [Technical Report for Ego4D Long-Term Action Anticipation Challenge 2025](https://mlsanity.com/paper/2506.02550/) (2025)
- [HCQA-1.5 @ Ego4D EgoSchema Challenge 2025](https://mlsanity.com/paper/2505.20644/) (2025)
- [OSGNet @ Ego4D Episodic Memory Challenge 2025](https://mlsanity.com/paper/2506.03710/) (2025)
- [Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challenges](https://mlsanity.com/paper/2511.13261/) (2025)
- [Four Eyes Are Better Than Two: Harnessing the Collaborative Potential of Large Models via Differentiated Thinking and Complementary Ensembles](https://mlsanity.com/paper/2505.16784/) (2025)
- [DIVE: Deep-search Iterative Video Exploration A Technical Report for the CVRR Challenge at CVPR 2025](https://mlsanity.com/paper/2506.21891/) (2025)
- [MARS: Technical Report for the CASTLE Challenge at EgoVis 2026](https://mlsanity.com/paper/2605.18176/) (2026)

## Abstract (arXiv)

In this report, we present our champion solution for Ego4D EgoSchema Challenge in CVPR 2024. To deeply integrate the powerful egocentric captioning model and question reasoning model, we propose a novel Hierarchical Comprehension scheme for egocentric video Question Answering, named HCQA. It consists of three stages: Fine-grained Caption Generation, Context-driven Summarization, and Inference-guided Answering. Given a long-form video, HCQA captures local detailed visual information and global summarised visual information via Fine-grained Caption Generation and Context-driven Summarization, respectively. Then in Inference-guided Answering, HCQA utilizes this hierarchical information to reason and answer given question. On the EgoSchema blind test set, HCQA achieves 75% accuracy in answering over 5,000 human curated multiple-choice questions. Our code will be released at https://github.com/Hyu-Zhang/HCQA.