# Transformation Driven Visual Reasoning

arXiv: https://arxiv.org/abs/2011.13160 · HTML: https://mlsanity.com/paper/2011.13160/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Nov 2020.

## Built on by (most cited first, 14 of 14)

- [StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objects](https://mlsanity.com/paper/2110.10189/) (2021)
- [Revisiting Open World Object Detection](https://mlsanity.com/paper/2201.00471/) (2022)
- [Visual Abductive Reasoning](https://mlsanity.com/paper/2203.14040/) (2022)
- [Explain Before You Answer: A Survey on Compositional Visual Reasoning](https://mlsanity.com/paper/2508.17298/) (2025)
- [Coarse-to-Fine Reasoning for Visual Question Answering](https://mlsanity.com/paper/2110.02526/) (2021)
- [VQA-GNN: Reasoning with Multimodal Knowledge via Graph Neural Networks for Visual Question Answering](https://mlsanity.com/paper/2205.11501/) (2022)
- [Reasoning about Actions over Visual and Linguistic Modalities: A Survey](https://mlsanity.com/paper/2207.07568/) (2022)
- [Benchmarks for Physical Reasoning AI](https://mlsanity.com/paper/2312.10728/) (2023)
- [Visual Transformation Telling](https://mlsanity.com/paper/2305.01928/) (2023)
- [Learning to Visually Connect Actions and their Effects](https://mlsanity.com/paper/2401.10805/) (2024)
- [ActionCOMET: A Zero-shot Approach to Learn Image-specific Commonsense Concepts about Actions](https://mlsanity.com/paper/2410.13662/) (2024)
- [Vision-Based Mistake Analysis in Procedural Activities: A Review of Advances and Challenges](https://mlsanity.com/paper/2510.19292/) (2025)
- [Understanding the Implicit User Intention via Reasoning with Large Language Model for Image Editing](https://mlsanity.com/paper/2510.27335/) (2025)
- [Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?](https://mlsanity.com/paper/2607.08514/) (2026)

## Abstract (arXiv)

This paper defines a new visual reasoning paradigm by introducing an important factor, i.e.~transformation. The motivation comes from the fact that most existing visual reasoning tasks, such as CLEVR in VQA, are solely defined to test how well the machine understands the concepts and relations within static settings, like one image. We argue that this kind of state driven visual reasoning approach has limitations in reflecting whether the machine has the ability to infer the dynamics between different states, which has been shown as important as state-level reasoning for human cognition in Piaget's theory. To tackle this problem, we propose a novel transformation driven visual reasoning task. Given both the initial and final states, the target is to infer the corresponding single-step or multi-step transformation, represented as a triplet (object, attribute, value) or a sequence of triplets, respectively. Following this definition, a new dataset namely TRANCE is constructed on the basis of CLEVR, including three levels of settings, i.e.~Basic (single-step transformation), Event (multi-step transformation), and View (multi-step transformation with variant views). Experimental results show that the state-of-the-art visual reasoning models perform well on Basic, but are still far from human-level intelligence on Event and View. We believe the proposed new paradigm will boost the development of machine visual reasoning. More advanced methods and real data need to be investigated in this direction. The resource of TVR is available at https://hongxin2019.github.io/TVR.