# GroundNLQ @ Ego4D Natural Language Queries Challenge 2023

arXiv: https://arxiv.org/abs/2306.15255 · HTML: https://mlsanity.com/paper/2306.15255/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 14 references that are in the map.
- First posted on arXiv 27 Jun 2023.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [ActionFormer: Localizing Moments of Actions with Transformers](https://mlsanity.com/paper/2202.07925/) (2022)
- [Span-based Localizing Network for Natural Language Video Localization](https://mlsanity.com/paper/2004.13931/) (2020)
- [InternVideo-Ego4D: A Pack of Champion Solutions to Ego4D Challenges](https://mlsanity.com/paper/2211.09529/) (2022)
- [NaQ: Leveraging Narrations as Queries to Supervise Episodic Memory](https://mlsanity.com/paper/2301.00746/) (2023)
- [ReLER@ZJU-Alibaba Submission to the Ego4D Natural Language Queries Challenge 2022](https://mlsanity.com/paper/2207.00383/) (2022)
- [Egocentric Video-Language Pretraining @ Ego4D Challenge 2022](https://mlsanity.com/paper/2207.01622/) (2022)
- [A Simple Transformer-Based Model for Ego4D Natural Language Queries Challenge](https://mlsanity.com/paper/2211.08704/) (2022)
- [An Efficient COarse-to-fiNE Alignment Framework @ Ego4D Natural Language Queries Challenge 2022](https://mlsanity.com/paper/2211.08776/) (2022)

## Built on by (most cited first, 20 of 20)

- [VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding](https://mlsanity.com/paper/2403.11481/) (2024)
- [Grounded Question-Answering in Long Egocentric Videos](https://mlsanity.com/paper/2312.06505/) (2023)
- [LifelongMemory: Leveraging LLMs for Answering Queries in Long-form Egocentric Videos](https://mlsanity.com/paper/2312.05269/) (2023)
- [EgoVideo: Exploring Egocentric Foundation Model and Downstream Adaptation](https://mlsanity.com/paper/2406.18070/) (2024)
- [Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topics](https://mlsanity.com/paper/2505.00675/) (2025)
- [Object-Shot Enhanced Grounding Network for Egocentric Video](https://mlsanity.com/paper/2505.04270/) (2025)
- [Evaluating Zero-Shot GPT-4V Performance on 3D Visual Question Answering Benchmarks](https://mlsanity.com/paper/2405.18831/) (2024)
- [ObjectNLQ @ Ego4D Episodic Memory Challenge 2024](https://mlsanity.com/paper/2406.15778/) (2024)
- [OmniQuery: Contextually Augmenting Captured Multimodal Memory to Enable Personal Question Answering](https://mlsanity.com/paper/2409.08250/) (2024)
- [ViLCo-Bench: VIdeo Language COntinual learning Benchmark](https://mlsanity.com/paper/2406.13123/) (2024)
- [OSGNet @ Ego4D Episodic Memory Challenge 2025](https://mlsanity.com/paper/2506.03710/) (2025)
- [EgoProx: Evaluating MLLMs on Egocentric 3D Proximity Reasoning Across a Cognitive Hierarchy](https://mlsanity.com/paper/2605.24456/) (2026)
- [CARLOR @ Ego4D Step Grounding Challenge: Bayesian temporal-order priors for test time refinement](https://mlsanity.com/paper/2406.09575/) (2024)
- [Streaming Detection of Queried Event Start](https://mlsanity.com/paper/2412.03567/) (2024)
- [GazeNLQ @ Ego4D Natural Language Queries Challenge 2025](https://mlsanity.com/paper/2506.05782/) (2025)
- [Incentivizing Temporal-Awareness in Egocentric Video Understanding Models](https://mlsanity.com/paper/2603.27184/) (2026)
- [Interactive Episodic Memory with User Feedback](https://mlsanity.com/paper/2604.24893/) (2026)
- [OSGNet with MLLM Reranking @ Ego4D Episodic Memory Challenge 2026](https://mlsanity.com/paper/2605.20818/) (2026)
- [Hand Trajectory Fusion for Egocentric Natural Language Query Grounding](https://mlsanity.com/paper/2606.02962/) (2026)
- [Linguistic Trajectory Encoding for Efficient Long-Horizon Spatial Memory in Embodied Agents](https://mlsanity.com/paper/2609.04802/) (2026)

## Abstract (arXiv)

In this report, we present our champion solution for Ego4D Natural Language Queries (NLQ) Challenge in CVPR 2023. Essentially, to accurately ground in a video, an effective egocentric feature extractor and a powerful grounding model are required. Motivated by this, we leverage a two-stage pre-training strategy to train egocentric feature extractors and the grounding model on video narrations, and further fine-tune the model on annotated data. In addition, we introduce a novel grounding model GroundNLQ, which employs a multi-modal multi-scale grounding module for effective video and text fusion and various temporal intervals, especially for long videos. On the blind test set, GroundNLQ achieves 25.67 and 18.18 for R1@IoU=0.3 and R1@IoU=0.5, respectively, and surpasses all other teams by a noticeable margin. Our code will be released aturlhttps://github.com/houzhijian/GroundNLQ.