BCAmirs at SemEval-2024 Task 4: Beyond Words: A Multimodal and Multilingual Exploration of Persuasion in Memes
Not on the map. Why?
Abstract (arXiv)
Memes, combining text and images, frequently use metaphors to convey persuasive messages, shaping public opinion. Motivated by this, our team engaged in SemEval-2024 Task 4, a hierarchical multi-label classification task designed to identify rhetorical and psychological persuasion techniques embedded within memes. To tackle this problem, we introduced a caption generation step to assess the modality gap and the impact of additional semantic information from images, which improved our result. Our best model utilizes GPT-4 generated captions alongside meme text to fine-tune RoBERTa as the text encoder and CLIP as the image encoder. It outperforms the baseline by a large margin in all 12 subtasks. In particular, it ranked in top-3 across all languages in Subtask 2a, and top-4 in Subtask 2b, demonstrating quantitatively strong performance. The improvement achieved by the introduced intermediate step is likely attributable to the metaphorical essence of images that challenges visual encoders. This highlights the potential for improving abstract visual semantics encoding.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 16 of its 33 references that are in the map. First posted on arXiv 3 Apr 2024.
Builds on
16 of 33 references.
References · 16
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
- 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
- 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
- 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
- 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,315
- 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
4 more
- 2021SemEval-2021 Task 6: Detection of Persuasion Techniques in Texts and Imagescited by 49
- 2023MemeCap: A Dataset for Captioning and Interpreting Memescited by 46
- 2023LM-CPPF: Paraphrasing-Guided Data Augmentation for Contrastive Prompt-Based Few-Shot Fine-Tuningcited by 15
- 2023Targeted Adversarial Attacks against Neural Machine Translationcited by 10
Built on by
Not cited yet.