# STAIR: Spatial-Temporal Reasoning with Auditable Intermediate Results for Video Question Answering

arXiv: https://arxiv.org/abs/2401.03901 · HTML: https://mlsanity.com/paper/2401.03901/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 1 of its 60 references that are in the map.
- First posted on arXiv 8 Jan 2024.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 1 of 1)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)

## Built on by (most cited first, 3 of 3)

- [ENTER: Event Based Interpretable Reasoning for VideoQA](https://mlsanity.com/paper/2501.14194/) (2025)
- [Evidence-Backed Video Question Answering](https://mlsanity.com/paper/2607.11862/) (2026)
- [See What You Need: Query-Aware Visual Intelligence through Reasoning-Perception Loops](https://mlsanity.com/paper/2508.17932/) (2025)

## Abstract (arXiv)

Recently we have witnessed the rapid development of video question answering models. However, most models can only handle simple videos in terms of temporal reasoning, and their performance tends to drop when answering temporal-reasoning questions on long and informative videos. To tackle this problem we propose STAIR, a Spatial-Temporal Reasoning model with Auditable Intermediate Results for video question answering. STAIR is a neural module network, which contains a program generator to decompose a given question into a hierarchical combination of several sub-tasks, and a set of lightweight neural modules to complete each of these sub-tasks. Though neural module networks are already widely studied on image-text tasks, applying them to videos is a non-trivial task, as reasoning on videos requires different abilities. In this paper, we define a set of basic video-text sub-tasks for video question answering and design a set of lightweight modules to complete them. Different from most prior works, modules of STAIR return intermediate outputs specific to their intentions instead of always returning attention maps, which makes it easier to interpret and collaborate with pre-trained models. We also introduce intermediate supervision to make these intermediate outputs more accurate. We conduct extensive experiments on several video question answering datasets under various settings to show STAIR's performance, explainability, compatibility with pre-trained models, and applicability when program annotations are not available. Code: https://github.com/yellow-binary-tree/STAIR