# LMBot: Distilling Graph Knowledge into Language Model for Graph-less Deployment in Twitter Bot Detection

arXiv: https://arxiv.org/abs/2306.17408 · HTML: https://mlsanity.com/paper/2306.17408/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 51 references that are in the map.
- First posted on arXiv 30 Jun 2023.

## Builds on (most cited first, 30 of 33)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [BERTweet: A pre-trained language model for English Tweets](https://mlsanity.com/paper/2005.10200/) (2020)
- [Towards Understanding and Mitigating Social Biases in Language Models](https://mlsanity.com/paper/2106.13219/) (2021)
- [Are we really making much progress? Revisiting, benchmarking, and refining heterogeneous graph neural networks](https://mlsanity.com/paper/2112.14936/) (2021)
- [Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation](https://mlsanity.com/paper/2110.08727/) (2021)
- [The Rise of Social Bots](https://mlsanity.com/paper/1407.5225/) (2014)
- [EDITS: Modeling and Mitigating Data Bias for Graph Neural Networks](https://mlsanity.com/paper/2108.05233/) (2021)
- [Scalable and Generalizable Social Bot Detection through Data Selection](https://mlsanity.com/paper/1911.09179/) (2019)
- [TwiBot-20: A Comprehensive Twitter Bot Detection Benchmark](https://mlsanity.com/paper/2106.13088/) (2021)
- [The paradigm-shift of social spambots: Evidence, theories, and tools for the arms race](https://mlsanity.com/paper/1701.03017/) (2017)
- [TwiBot-22: Towards Graph-Based Twitter Bot Detection](https://mlsanity.com/paper/2206.04564/) (2022)
- [Heterogeneity-aware Twitter Bot Detection with Relational Graph Transformers](https://mlsanity.com/paper/2109.02927/) (2021)
- [BotRGCN: Twitter Bot Detection with Relational Graph Convolutional Networks](https://mlsanity.com/paper/2106.13092/) (2021)
- [Linkless Link Prediction via Relational Distillation](https://mlsanity.com/paper/2210.05801/) (2022)
- [Deep Neural Networks for Bot Detection](https://mlsanity.com/paper/1802.04289/) (2018)
- [Fame for sale: efficient detection of fake Twitter followers](https://mlsanity.com/paper/1509.04098/) (2015)
- [Twitter Bot Detection Using Bidirectional Long Short-term Memory Neural Networks and Word Embeddings](https://mlsanity.com/paper/2002.01336/) (2020)
- [SATAR: A Self-supervised Approach to Twitter Account Representation Learning and its Application in Bot Detection](https://mlsanity.com/paper/2106.13089/) (2021)
- [BotMoE: Twitter Bot Detection with Community-Aware Mixtures of Modal-Specific Experts](https://mlsanity.com/paper/2304.06280/) (2023)
- [RoSGAS: Adaptive Social Bot Detection with Reinforced Self-Supervised GNN Architecture Search](https://mlsanity.com/paper/2206.06757/) (2022)
- [BIC: Twitter Bot Detection with Text-Graph Interaction and Semantic Consistency](https://mlsanity.com/paper/2208.08320/) (2022)

## Built on by (most cited first, 17 of 17)

- [Large Language Models for Cyber Security: A Systematic Literature Review](https://mlsanity.com/paper/2405.04760/) (2024)
- [What Does the Bot Say? Opportunities and Risks of Large Language Models in Social Media Bot Detection](https://mlsanity.com/paper/2402.00371/) (2024)
- [Leveraging Large Language Models to Detect Influence Campaigns in Social Media](https://mlsanity.com/paper/2311.07816/) (2023)
- [GuARD: Effective Anomaly Detection through a Text-Rich and Graph-Informed Language Model](https://mlsanity.com/paper/2412.03930/) (2024)
- [Large Language Models Meet Text-Attributed Graphs: A Survey of Integration Frameworks and Applications](https://mlsanity.com/paper/2510.21131/) (2025)
- [Unmasking Social Bots: How Confident Are We?](https://mlsanity.com/paper/2407.13929/) (2024)
- [EUREKHA: Enhancing User Representation for Key Hackers Identification in Underground Forums](https://mlsanity.com/paper/2411.05479/) (2024)
- [TRACE-Bot: Detecting Emerging LLM-Driven Social Bots via Implicit Semantic Representations and AIGC-Enhanced Behavioral Patterns](https://mlsanity.com/paper/2604.02147/) (2026)
- [RABot: Reinforcement-Guided Graph Augmentation for Imbalanced and Noisy Social Bot Detection](https://mlsanity.com/paper/2602.21749/) (2026)
- [FedRio: Personalized Federated Social Bot Detection via Cooperative Reinforced Contrastive Adversarial Distillation](https://mlsanity.com/paper/2604.10678/) (2026)
- [Unveiling the Hidden: Movie Genre and User Bias in Spoiler Detection](https://mlsanity.com/paper/2504.17834/) (2025)
- [Masked Language Models are Good Heterogeneous Graph Generalizers](https://mlsanity.com/paper/2506.06157/) (2025)
- [Large Language Models in Cybersecurity: Applications, Vulnerabilities, and Defense Techniques](https://mlsanity.com/paper/2507.13629/) (2025)
- [Bot Meets Shortcut: How Can LLMs Aid in Handling Unknown Invariance OOD Scenarios?](https://mlsanity.com/paper/2511.08455/) (2025)
- [SAHG: Sector-Anisotropic Hyperbolic Graph Model for Social Bot Detection](https://mlsanity.com/paper/2605.30166/) (2026)
- [Breaking and Defending LLM-Powered Social Media Bot Detection Systems](https://mlsanity.com/paper/2608.15893/) (2026)
- [FISSION: Label Augmentation for Bot Detection](https://mlsanity.com/paper/2609.26279/) (2026)

## Abstract (arXiv)

As malicious actors employ increasingly advanced and widespread bots to disseminate misinformation and manipulate public opinion, the detection of Twitter bots has become a crucial task. Though graph-based Twitter bot detection methods achieve state-of-the-art performance, we find that their inference depends on the neighbor users multi-hop away from the targets, and fetching neighbors is time-consuming and may introduce bias. At the same time, we find that after finetuning on Twitter bot detection, pretrained language models achieve competitive performance and do not require a graph structure during deployment. Inspired by this finding, we propose a novel bot detection framework LMBot that distills the knowledge of graph neural networks (GNNs) into language models (LMs) for graph-less deployment in Twitter bot detection to combat the challenge of data dependency. Moreover, LMBot is compatible with graph-based and graph-less datasets. Specifically, we first represent each user as a textual sequence and feed them into the LM for domain adaptation. For graph-based datasets, the output of LMs provides input features for the GNN, enabling it to optimize for bot detection and distill knowledge back to the LM in an iterative, mutually enhancing process. Armed with the LM, we can perform graph-less inference, which resolves the graph data dependency and sampling bias issues. For datasets without graph structure, we simply replace the GNN with an MLP, which has also shown strong performance. Our experiments demonstrate that LMBot achieves state-of-the-art performance on four Twitter bot detection benchmarks. Extensive studies also show that LMBot is more robust, versatile, and efficient compared to graph-based Twitter bot detection methods.