GEST: the Graph of Events in Space and Time as a Common Representation between Vision and Language
Not on the map. Why?
Abstract (arXiv)
One of the essential human skills is the ability to seamlessly build an inner representation of the world. By exploiting this representation, humans are capable of easily finding consensus between visual, auditory and linguistic perspectives. In this work, we set out to understand and emulate this ability through an explicit representation for both vision and language - Graphs of Events in Space and Time (GEST). GEST alows us to measure the similarity between texts and videos in a semantic and fully explainable way, through graph matching. It also allows us to generate text and videos from a common representation that provides a well understood content. In this work we show that the graph matching similarity metrics based on GEST outperform classical text generation metrics and can also boost the performance of state of art, heavily trained metrics.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 34 of its 58 references that are in the map. First posted on arXiv 22 May 2023.
Builds on
34 of 58 references.
References · 34
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
- 2019A Comprehensive Survey on Graph Neural Networkscited by 2,906
- 2015VQA: Visual Question Answeringcited by 2,489
- 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
- 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
- 2018Graph Neural Networks: A Review of Methods and Applicationscited by 1,571
- 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
- 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
22 more
- 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
- 2016Generative Adversarial Text to Image Synthesiscited by 735
- 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
- 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 309
- 2016Hierarchical Question-Image Co-Attention for Visual Question Answeringcited by 278
- 2016Image Captioning with Semantic Attentioncited by 260
- 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
- 2018Videos as Space-Time Region Graphscited by 221
- 2018End-to-End Dense Video Captioning with Masked Transformercited by 214
- 2020A Survey of Evaluation Metrics Used for NLG Systemscited by 164
- 2017Video Generation From Textcited by 153
- 2018RSA: Byzantine-Robust Stochastic Aggregation Methods for Distributed Learning from Heterogeneous Datasetscited by 148
- 2020Investigating Pretrained Language Models for Graph-to-Text Generationcited by 117
- 2018Reconstruction Network for Video Captioningcited by 98
- 2017Semantic Image Synthesis via Adversarial Learningcited by 87
- 2019Neural Graph Matching Network: Learning Lawler's Quadratic Assignment Problem with Extension to Hypergraph and Multiple-graph Matchingcited by 83
- 2017Convolutional Image Captioningcited by 52
- 2018Commonsense for Generative Multi-Hop Question Answering Taskscited by 48
- 2022(2.5+1)D Spatio-Temporal Scene Graphs for Video Question Answeringcited by 29
- 2017Temporal Dynamic Graph LSTM for Action-driven Video Object Detectioncited by 23
- 2019Text Guided Person Image Synthesiscited by 22
- 2016Efficient Activity Detection in Untrimmed Video with Max-Subgraph Searchcited by 1
Built on by
Not cited yet.