# Modeling Temporal-Modal Entity Graph for Procedural Multimodal Machine Comprehension

arXiv: https://arxiv.org/abs/2204.02566 · HTML: https://mlsanity.com/paper/2204.02566/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 40 references that are in the map.
- First posted on arXiv 6 Apr 2022.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [MovieQA: Understanding Stories in Movies through Question-Answering](https://mlsanity.com/paper/1512.02902/) (2015)
- [A Multiscale Visualization of Attention in the Transformer Model](https://mlsanity.com/paper/1906.05714/) (2019)
- [A Fast and Accurate One-Stage Approach to Visual Grounding](https://mlsanity.com/paper/1908.06354/) (2019)
- [RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipes](https://mlsanity.com/paper/1809.00812/) (2018)
- [Connecting the Dots: Document-level Neural Relation Extraction with Edge-oriented Graphs](https://mlsanity.com/paper/1909.00228/) (2019)
- [Simulating Action Dynamics with Neural Process Networks](https://mlsanity.com/paper/1711.05313/) (2017)
- [A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translation](https://mlsanity.com/paper/2007.08742/) (2020)
- [The Amazing Mysteries of the Gutter: Drawing Inferences Between Panels in Comic Book Narratives](https://mlsanity.com/paper/1611.05118/) (2016)
- [A Dataset for Tracking Entities in Open Domain Procedural Text](https://mlsanity.com/paper/2011.08092/) (2020)
- [Everything Happens for a Reason: Discovering the Purpose of Actions in Procedural Text](https://mlsanity.com/paper/1909.04745/) (2019)
- [Time-Stamped Language Model: Teaching Language Models to Understand the Flow of Events](https://mlsanity.com/paper/2104.07635/) (2021)
- [Transformer Reasoning Network for Image-Text Matching and Retrieval](https://mlsanity.com/paper/2004.09144/) (2020)
- [Building Dynamic Knowledge Graphs from Text using Machine Reading Comprehension](https://mlsanity.com/paper/1810.05682/) (2018)
- [Knowledge-Aware Procedural Text Understanding with Multi-Stage Training](https://mlsanity.com/paper/2009.13199/) (2020)
- [Procedural Reading Comprehension with Attribute-Aware Context Flow](https://mlsanity.com/paper/2003.13878/) (2020)
- [A Recipe for Creating Multimodal Aligned Datasets for Sequential Tasks](https://mlsanity.com/paper/2005.09606/) (2020)
- [A Benchmark for Structured Procedural Knowledge Extraction from Cooking Videos](https://mlsanity.com/paper/2005.00706/) (2020)
- [Unsupervised Discovery of Multimodal Links in Multi-image, Multi-sentence Documents](https://mlsanity.com/paper/1904.07826/) (2019)
- [Procedural Reasoning Networks for Understanding Multimodal Procedures](https://mlsanity.com/paper/1909.08859/) (2019)
- [Cross-Modal Coherence for Text-to-Image Retrieval](https://mlsanity.com/paper/2109.11047/) (2021)
- [A Heterogeneous Graph with Factual, Temporal and Logical Knowledge for Question Answering Over Dynamic Contexts](https://mlsanity.com/paper/2004.12057/) (2020)

## Built on by (most cited first, 2 of 2)

- [Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey](https://mlsanity.com/paper/2411.17558/) (2024)
- [Multimodal Inverse Cloze Task for Knowledge-based Visual Question Answering](https://mlsanity.com/paper/2301.04366/) (2023)

## Abstract (arXiv)

Procedural Multimodal Documents (PMDs) organize textual instructions and corresponding images step by step. Comprehending PMDs and inducing their representations for the downstream reasoning tasks is designated as Procedural MultiModal Machine Comprehension (M3C). In this study, we approach Procedural M3C at a fine-grained level (compared with existing explorations at a document or sentence level), that is, entity. With delicate consideration, we model entity both in its temporal and cross-modal relation and propose a novel Temporal-Modal Entity Graph (TMEG). Specifically, graph structure is formulated to capture textual and visual entities and trace their temporal-modal evolution. In addition, a graph aggregation module is introduced to conduct graph encoding and reasoning. Comprehensive experiments across three Procedural M3C tasks are conducted on a traditional dataset RecipeQA and our new dataset CraftQA, which can better evaluate the generalization of TMEG.