mlsanityData to 7 Oct 2026

Reasoning about Actions over Visual and Linguistic Modalities: A Survey

arXiv 2207.07568 ↗First posted 15 Jul 2022cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

'Actions' play a vital role in how humans interact with the world and enable them to achieve desired goals. As a result, most common sense (CS) knowledge for humans revolves around actions. While 'Reasoning about Actions & Change' (RAC) has been widely studied in the Knowledge Representation community, it has recently piqued the interest of NLP and computer vision researchers. This paper surveys existing tasks, benchmark datasets, various techniques and models, and their respective performance concerning advancements in RAC in the vision and language domain. Towards the end, we summarize our key takeaways, discuss the present challenges facing this research area, and outline potential directions for future research.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 32 of its 52 references that are in the map. First posted on arXiv 15 Jul 2022.

Builds on

32 of 52 references.

References · 32

  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  2. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  3. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  4. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  5. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  6. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  7. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
  8. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  9. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 309
  10. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
  11. 2018Composing Text and Image for Image Retrieval - An Empirical Odysseycited by 234
  12. 2018Learning to Describe Differences Between Pairs of Similar Imagescited by 114
20 more
  1. 2019Procedure Planning in Instructional Videoscited by 92
  2. 2021A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Executioncited by 91
  3. 2018Vision-based Navigation with Language-based Assistance via Imitation Learning with Indirect Interventioncited by 77
  4. 2018RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipescited by 74
  5. 2021Learning to Compose Visual Relationscited by 72
  6. 2019WIQA: A dataset for "What if..." reasoning over procedural textcited by 71
  7. 2021Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoningcited by 68
  8. 2020Attention over learned object embeddings enables complex visual reasoningcited by 59
  9. 2021Visual Goal-Step Inference using wikiHowcited by 42
  10. 2019Everything Happens for a Reason: Discovering the Purpose of Actions in Procedural Textcited by 31
  11. 2017Verb Physics: Relative Physical Knowledge of Actions and Objectscited by 26
  12. 2021Tiered Reasoning for Intuitive Physics: Toward Verifiable Commonsense Language Understandingcited by 19
  13. 2017Learning Interpretable Spatial Operations in a Rich 3D Blocks Worldcited by 15
  14. 2020Learning About Objects by Learning to Interact with Themcited by 14
  15. 2020Graph Edit Distance Reward: Learning to Edit Scene Graphcited by 14
  16. 2020Transformation Driven Visual Reasoningcited by 14
  17. 2021CLEVR_HYP: A Challenge Dataset and Baselines for Visual Question Answering with Hypothetical Actions over Imagescited by 14
  18. 2018Answering Visual What-If Questions: From Actions to Predicted Scene Descriptionscited by 10
  19. 2019Pre-Learning Environment Representations for Data-Efficient Neural Instruction Followingcited by 6
  20. 2020Following Instructions by Imagining and Reaching Visual Goalscited by 6

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 5; 2024: 1; 2025: 1.

10 papers.

  1. 2023Large Language Models are Visual Reasoning Coordinatorscited by 58
  2. 2022CRIPP-VQA: Counterfactual Reasoning about Implicit Physical Properties via Video Question Answeringcited by 13
  3. 2025Perception in Reflectioncited by 13
  4. 2024Hybrid Reasoning Based on Large Language Models for Autonomous Car Drivingcited by 11
  5. 2023MuG: A Multimodal Classification Benchmark on Game Data with Tabular, Textual, and Visual Fieldscited by 5
  6. 2023A Survey on Image-text Multimodal Modelscited by 4
  7. 2022Learning Action-Effect Dynamics for Hypothetical Vision-Language Reasoning Taskcited by 2
  8. 2023A Survey on Interpretable Cross-modal Reasoningcited by 2
  9. 2023Implicit Affordance Acquisition via Causal Action-Effect Modeling in the Video Domaincited by 1
  10. 2022Learning Action-Effect Dynamics from Pairs of Scene-graphscited by 0