Ring That Bell: A Corpus and Method for Multimodal Metaphor Detection in Videos
Not on the map. Why?
Abstract (arXiv)
We present the first openly available multimodal metaphor annotated corpus. The corpus consists of videos including audio and subtitles that have been annotated by experts. Furthermore, we present a method for detecting metaphors in the new dataset based on the textual content of the videos. The method achieves a high F1-score (62%) for metaphorical labels. We also experiment with other modalities and multimodal methods; however, these methods did not out-perform the text-based model. In our error analysis, we do identify that there are cases where video could help in disambiguating metaphors, however, the visual cues are too subtle for our model to capture. The data is available on Zenodo.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 18 of its 55 references that are in the map. First posted on arXiv 15 Dec 2022.
Builds on
18 of 55 references.
References · 18
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
- 2017The Kinetics Human Action Video Datasetcited by 1,894
- 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
- 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
- 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
- 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
- 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 269
- 2019Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)cited by 173
- 2018Memory Fusion Network for Multi-view Sequential Learningcited by 167
- 2019M3ER: Multiplicative Multimodal Emotion Recognition Using Facial, Textual, and Speech Cuescited by 56
6 more
- 2019Modality to Modality Translation: An Adversarial Representation Learning and Graph Fusion Network for Multimodal Fusioncited by 45
- 2018Neural Metaphor Detection in Contextcited by 34
- 2021Quantum-inspired Multimodal Fusion for Video Sentiment Analysiscited by 13
- 2019Let's FACE it. Finnish Poetry Generation with Aesthetics and Framingcited by 11
- 2020"Love is as Complex as Math": Metaphor Generation System for Social Chatbotcited by 9
- 2021!Qu'e maravilla! Multimodal Sarcasm Detection in Spanish: a Dataset and a Baselinecited by 8
Built on by
Citing papers per year: 2025: 2; 2026: 1.
3 papers.