# Ring That Bell: A Corpus and Method for Multimodal Metaphor Detection in Videos

arXiv: https://arxiv.org/abs/2301.01134 · HTML: https://mlsanity.com/paper/2301.01134/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 55 references that are in the map.
- First posted on arXiv 15 Dec 2022.

## Builds on (most cited first, 18 of 18)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [MLS: A Large-Scale Multilingual Dataset for Speech Research](https://mlsanity.com/paper/2012.03411/) (2020)
- [Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection](https://mlsanity.com/paper/2004.10643/) (2020)
- [Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)](https://mlsanity.com/paper/1906.01815/) (2019)
- [Memory Fusion Network for Multi-view Sequential Learning](https://mlsanity.com/paper/1802.00927/) (2018)
- [M3ER: Multiplicative Multimodal Emotion Recognition Using Facial, Textual, and Speech Cues](https://mlsanity.com/paper/1911.05659/) (2019)
- [Modality to Modality Translation: An Adversarial Representation Learning and Graph Fusion Network for Multimodal Fusion](https://mlsanity.com/paper/1911.07848/) (2019)
- [Neural Metaphor Detection in Context](https://mlsanity.com/paper/1808.09653/) (2018)
- [Quantum-inspired Multimodal Fusion for Video Sentiment Analysis](https://mlsanity.com/paper/2103.10572/) (2021)
- [Let's FACE it. Finnish Poetry Generation with Aesthetics and Framing](https://mlsanity.com/paper/1910.13946/) (2019)
- ["Love is as Complex as Math": Metaphor Generation System for Social Chatbot](https://mlsanity.com/paper/2001.00733/) (2020)
- [!Qu'e maravilla! Multimodal Sarcasm Detection in Spanish: a Dataset and a Baseline](https://mlsanity.com/paper/2105.05542/) (2021)

## Built on by (most cited first, 3 of 3)

- [Looking Beyond the Obvious: A Survey on Abstract Concept Recognition for Video Understanding](https://mlsanity.com/paper/2508.20765/) (2025)
- [Hummus: A Dataset of Humorous Multimodal Metaphor Use](https://mlsanity.com/paper/2504.02983/) (2025)
- [MetaphorVU: Towards Metaphorical Video Understanding](https://mlsanity.com/paper/2605.25461/) (2026)

## Abstract (arXiv)

We present the first openly available multimodal metaphor annotated corpus. The corpus consists of videos including audio and subtitles that have been annotated by experts. Furthermore, we present a method for detecting metaphors in the new dataset based on the textual content of the videos. The method achieves a high F1-score (62%) for metaphorical labels. We also experiment with other modalities and multimodal methods; however, these methods did not out-perform the text-based model. In our error analysis, we do identify that there are cases where video could help in disambiguating metaphors, however, the visual cues are too subtle for our model to capture. The data is available on Zenodo.