# Multimodal Detection of Bots on X (Twitter) using Transformers

arXiv: https://arxiv.org/abs/2308.14484 · HTML: https://mlsanity.com/paper/2308.14484/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 68 references that are in the map.
- First posted on arXiv 28 Aug 2023.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient](https://mlsanity.com/paper/1609.05473/) (2016)
- [One weird trick for parallelizing convolutional neural networks](https://mlsanity.com/paper/1404.5997/) (2014)
- [The spread of low-credibility content by social bots](https://mlsanity.com/paper/1707.07592/) (2017)
- [Scalable and Generalizable Social Bot Detection through Data Selection](https://mlsanity.com/paper/1911.09179/) (2019)
- [TwiBot-20: A Comprehensive Twitter Bot Detection Benchmark](https://mlsanity.com/paper/2106.13088/) (2021)
- [The paradigm-shift of social spambots: Evidence, theories, and tools for the arms race](https://mlsanity.com/paper/1701.03017/) (2017)
- [TwHIN-BERT: A Socially-Enriched Pre-trained Language Model for Multilingual Tweet Representations at Twitter](https://mlsanity.com/paper/2209.07562/) (2022)
- [BotOrNot: A System to Evaluate Social Bots](https://mlsanity.com/paper/1602.00975/) (2016)
- [Heterogeneity-aware Twitter Bot Detection with Relational Graph Transformers](https://mlsanity.com/paper/2109.02927/) (2021)
- [BotRGCN: Twitter Bot Detection with Relational Graph Convolutional Networks](https://mlsanity.com/paper/2106.13092/) (2021)
- [Deep Neural Networks for Bot Detection](https://mlsanity.com/paper/1802.04289/) (2018)
- [Twitter Bot Detection Using Bidirectional Long Short-term Memory Neural Networks and Word Embeddings](https://mlsanity.com/paper/2002.01336/) (2020)
- [RoSGAS: Adaptive Social Bot Detection with Reinforced Self-Supervised GNN Architecture Search](https://mlsanity.com/paper/2206.06757/) (2022)
- [DNA-inspired online behavioral modeling and its application to spambot detection](https://mlsanity.com/paper/1602.00110/) (2016)
- [BIC: Twitter Bot Detection with Text-Graph Interaction and Semantic Consistency](https://mlsanity.com/paper/2208.08320/) (2022)
- [Detecting Dementia from Speech and Transcripts using Transformers](https://mlsanity.com/paper/2110.14769/) (2021)
- [Calibration of Transformer-based Models for Identifying Stress and Depression in Social Media](https://mlsanity.com/paper/2305.16797/) (2023)
- [Point-of-Interest Type Prediction using Text and Images](https://mlsanity.com/paper/2109.00602/) (2021)

## Built on by (most cited first, 2 of 2)

- [A Graph Neural Architecture Search Approach for Identifying Bots in Social Media](https://mlsanity.com/paper/2411.16285/) (2024)
- [Graph-based Fake Account Detection: A Survey](https://mlsanity.com/paper/2507.06541/) (2025)

## Abstract (arXiv)

Although not all bots are malicious, the vast majority of them are responsible for spreading misinformation and manipulating the public opinion about several issues, i.e., elections and many more. Therefore, the early detection of bots is crucial. Although there have been proposed methods for detecting bots in social media, there are still substantial limitations. For instance, existing research initiatives still extract a large number of features and train traditional machine learning algorithms or use GloVe embeddings and train LSTMs. However, feature extraction is a tedious procedure demanding domain expertise. Also, language models based on transformers have been proved to be better than LSTMs. Other approaches create large graphs and train graph neural networks requiring in this way many hours for training and access to computational resources. To tackle these limitations, this is the first study employing only the user description field and images of three channels denoting the type and content of tweets posted by the users. Firstly, we create digital DNA sequences, transform them to 3d images, and apply pretrained models of the vision domain, including EfficientNet, AlexNet, VGG16, etc. Next, we propose a multimodal approach, where we use TwHIN-BERT for getting the textual representation of the user description field and employ VGG16 for acquiring the visual representation for the image modality. We propose three different fusion methods, namely concatenation, gated multimodal unit, and crossmodal attention, for fusing the different modalities and compare their performances. Finally, we present a qualitative analysis of the behavior of our best performing model. Extensive experiments conducted on the Cresci'17 and TwiBot-20 datasets demonstrate valuable advantages of our introduced approaches over state-of-the-art ones.