# Exploiting BERT For Multimodal Target Sentiment Classification Through Input Space Translation

arXiv: https://arxiv.org/abs/2108.01682 · HTML: https://mlsanity.com/paper/2108.01682/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 51 references that are in the map.
- First posted on arXiv 3 Aug 2021.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [BERTweet: A pre-trained language model for English Tweets](https://mlsanity.com/paper/2005.10200/) (2020)
- [MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2005.03545/) (2020)
- [MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos](https://mlsanity.com/paper/1606.06259/) (2016)
- [BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis](https://mlsanity.com/paper/1904.02232/) (2019)
- [Pretrained Transformers as Universal Computation Engines](https://mlsanity.com/paper/2103.05247/) (2021)
- [Challenges in Deploying Machine Learning: a Survey of Case Studies](https://mlsanity.com/paper/2011.09926/) (2020)
- [Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities](https://mlsanity.com/paper/1812.07809/) (2018)
- [Utilizing BERT for Aspect-Based Sentiment Analysis via Constructing Auxiliary Sentence](https://mlsanity.com/paper/1903.09588/) (2019)
- [Do Transformer Modifications Transfer Across Implementations and Applications?](https://mlsanity.com/paper/2102.11972/) (2021)
- [Beneath the Tip of the Iceberg: Current Challenges and New Directions in Sentiment Analysis Research](https://mlsanity.com/paper/2005.00357/) (2020)
- [Exploiting BERT for End-to-End Aspect-based Sentiment Analysis](https://mlsanity.com/paper/1910.00883/) (2019)
- [M3ER: Multiplicative Multimodal Emotion Recognition Using Facial, Textual, and Speech Cues](https://mlsanity.com/paper/1911.05659/) (2019)
- [Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods](https://mlsanity.com/paper/1907.09358/) (2019)
- [Transform and Tell: Entity-Aware News Image Captioning](https://mlsanity.com/paper/2004.08070/) (2020)
- [Length-Controllable Image Captioning](https://mlsanity.com/paper/2007.09580/) (2020)
- [Visual Question Answering as Reading Comprehension](https://mlsanity.com/paper/1811.11903/) (2018)
- [Reproducibility, Replicability and Beyond: Assessing Production Readiness of Aspect Based Sentiment Analysis in the Wild](https://mlsanity.com/paper/2101.09449/) (2021)
- [Mining fine-grained opinions on closed captions of YouTube videos with an attention-RNN](https://mlsanity.com/paper/1708.02420/) (2017)

## Built on by (most cited first, 30 of 41)

- [Towards Adversarial Attack on Vision-Language Pre-training Models](https://mlsanity.com/paper/2206.09391/) (2022)
- [Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models](https://mlsanity.com/paper/2307.14061/) (2023)
- [AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learning](https://mlsanity.com/paper/2308.07026/) (2023)
- [A Survey on Aspect-Based Sentiment Analysis: Tasks, Methods, and Challenges](https://mlsanity.com/paper/2203.01054/) (2022)
- [Boosting Transferability in Vision-Language Attacks via Diversification along the Intersection Region of Adversarial Trajectory](https://mlsanity.com/paper/2403.12445/) (2024)
- [A Survey on Multi-modal Summarization](https://mlsanity.com/paper/2109.05199/) (2021)
- [Vision-Language Pre-Training for Multimodal Aspect-Based Sentiment Analysis](https://mlsanity.com/paper/2204.07955/) (2022)
- [Counterfactual Reasoning for Out-of-distribution Multimodal Sentiment Analysis](https://mlsanity.com/paper/2207.11652/) (2022)
- [Semantic-Aligned Adversarial Evolution Triangle for High-Transferability Vision-Language Attack](https://mlsanity.com/paper/2411.02669/) (2024)
- [WisdoM: Improving Multimodal Sentiment Analysis by Fusing Contextual World Knowledge](https://mlsanity.com/paper/2401.06659/) (2024)
- [Learning to Rebalance Multi-Modal Optimization by Adaptively Masking Subnetworks](https://mlsanity.com/paper/2404.08347/) (2024)
- [Large Language Models Meet Text-Centric Multimodal Sentiment Analysis: A Survey](https://mlsanity.com/paper/2406.08068/) (2024)
- [Few-shot Multimodal Sentiment Analysis based on Multimodal Probabilistic Fusion Prompts](https://mlsanity.com/paper/2211.06607/) (2022)
- [AoM: Detecting Aspect-oriented Information for Multimodal Aspect-Based Sentiment Analysis](https://mlsanity.com/paper/2306.01004/) (2023)
- [M2DF: Multi-grained Multi-curriculum Denoising Framework for Multimodal Aspect-based Sentiment Analysis](https://mlsanity.com/paper/2310.14605/) (2023)
- [Exploring Cognitive and Aesthetic Causality for Multimodal Aspect-Based Sentiment Analysis](https://mlsanity.com/paper/2504.15848/) (2025)
- [MACSA: A Multimodal Aspect-Category Sentiment Analysis Dataset with Multimodal Fine-grained Aligned Annotations](https://mlsanity.com/paper/2206.13969/) (2022)
- [A Novel Energy based Model Mechanism for Multi-modal Aspect-Based Sentiment Analysis](https://mlsanity.com/paper/2312.08084/) (2023)
- [Target-oriented Sentiment Classification with Sequential Cross-modal Semantic Graph](https://mlsanity.com/paper/2208.09417/) (2022)
- [The Emotions of the Crowd: Learning Image Sentiment from Tweets via Cross-modal Distillation](https://mlsanity.com/paper/2304.14942/) (2023)
- [RethinkingTMSC: An Empirical Study for Target-Oriented Multimodal Sentiment Classification](https://mlsanity.com/paper/2310.09596/) (2023)
- [Impact of Stickers on Multimodal Sentiment and Intent in Social Media: A New Task, Dataset and Baseline](https://mlsanity.com/paper/2405.08427/) (2024)
- [Enhanced Multimodal Aspect-Based Sentiment Analysis by LLM-Generated Rationales](https://mlsanity.com/paper/2505.14499/) (2025)
- [MemeTAG: Keyword-Driven Meme Classification through Tag Embedding Reconstruction](https://mlsanity.com/paper/2609.20962/) (2026)
- [Bridging the Emotional Semantic Gap via Multimodal Relevance Estimation](https://mlsanity.com/paper/2302.01555/) (2023)
- [CorMulT: A Semi-supervised Modality Correlation-aware Multimodal Transformer for Sentiment Analysis](https://mlsanity.com/paper/2407.07046/) (2024)
- [Dependency Structure Augmented Contextual Scoping Framework for Multimodal Aspect-Based Sentiment Analysis](https://mlsanity.com/paper/2504.11331/) (2025)
- [Robust Multimodal Sentiment Analysis of Image-Text Pairs by Distribution-Based Feature Recovery and Fusion](https://mlsanity.com/paper/2511.18751/) (2025)
- [MEMEWEAVER: Inter-Meme Graph Reasoning for Sexism and Misogyny Detection](https://mlsanity.com/paper/2601.08684/) (2026)
- [VAuLT: Augmenting the Vision-and-Language Transformer for Sentiment Classification on Social Media](https://mlsanity.com/paper/2208.09021/) (2022)

## Abstract (arXiv)

Multimodal target/aspect sentiment classification combines multimodal sentiment analysis and aspect/target sentiment classification. The goal of the task is to combine vision and language to understand the sentiment towards a target entity in a sentence. Twitter is an ideal setting for the task because it is inherently multimodal, highly emotional, and affects real world events. However, multimodal tweets are short and accompanied by complex, possibly irrelevant images. We introduce a two-stream model that translates images in input space using an object-aware transformer followed by a single-pass non-autoregressive text generation approach. We then leverage the translation to construct an auxiliary sentence that provides multimodal information to a language model. Our approach increases the amount of text available to the language model and distills the object-level information in complex images. We achieve state-of-the-art performance on two multimodal Twitter datasets without modifying the internals of the language model to accept multimodal data, demonstrating the effectiveness of our translation. In addition, we explain a failure mode of a popular approach for aspect sentiment analysis when applied to tweets. Our code is available at textcolorblueurlhttps://github.com/codezakh/exploiting-BERT-thru-translation.