# Winning the CVPR'2022 AQTC Challenge: A Two-stage Function-centric Approach

arXiv: https://arxiv.org/abs/2206.09597 · HTML: https://mlsanity.com/paper/2206.09597/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 22 references that are in the map.
- First posted on arXiv 20 Jun 2022.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [Visual Dialog](https://mlsanity.com/paper/1611.08669/) (2016)
- [Bilinear Attention Networks](https://mlsanity.com/paper/1805.07932/) (2018)
- [Stacked Attention Networks for Image Question Answering](https://mlsanity.com/paper/1511.02274/) (2015)
- [Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding](https://mlsanity.com/paper/1606.01847/) (2016)
- [AssistQ: Affordance-centric Question-driven Task Completion for Egocentric Assistant](https://mlsanity.com/paper/2203.04203/) (2022)
- [Unified Learning Approach for Egocentric Hand Gesture Recognition and Fingertip Detection](https://mlsanity.com/paper/2101.02047/) (2021)

## Built on by (most cited first, 3 of 3)

- [VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computation](https://mlsanity.com/paper/2408.16730/) (2024)
- [First Place Solution to the CVPR'2023 AQTC Challenge: A Function-Interaction Centric Approach with Spatiotemporal Visual-Language Alignment](https://mlsanity.com/paper/2306.13380/) (2023)
- [A Solution to CVPR'2023 AQTC Challenge: Video Alignment for Multi-Step Inference](https://mlsanity.com/paper/2306.14412/) (2023)

## Abstract (arXiv)

Affordance-centric Question-driven Task Completion for Egocentric Assistant(AQTC) is a novel task which helps AI assistant learn from instructional videos and scripts and guide the user step-by-step. In this paper, we deal with the AQTC via a two-stage Function-centric approach, which consists of Question2Function Module to ground the question with the related function and Function2Answer Module to predict the action based on the historical steps. We evaluated several possible solutions in each module and obtained significant gains compared to the given baselines. Our code is available at urlhttps://github.com/starsholic/LOVEU-CVPR22-AQTC.