mlsanityData to 7 Oct 2026

TwHIN-BERT: A Socially-Enriched Pre-trained Language Model for Multilingual Tweet Representations at Twitter

arXiv 2209.07562 ↗First posted 15 Sep 2022cs.CLcited by 44

Not on the map. Why?

Abstract (arXiv)

Pre-trained language models (PLMs) are fundamental for natural language processing applications. Most existing PLMs are not tailored to the noisy user-generated text on social media, and the pre-training does not factor in the valuable social engagement logs available in a social network. We present TwHIN-BERT, a multilingual language model productionized at Twitter, trained on in-domain data from the popular social network. TwHIN-BERT differs from prior pre-trained language models as it is trained with not only text-based self-supervision, but also with a social objective based on the rich social engagements within a Twitter heterogeneous information network (TwHIN). Our model is trained on 7 billion tweets covering over 100 distinct languages, providing a valuable representation to model short, noisy, user-generated text. We evaluate our model on various multilingual social recommendation and semantic understanding tasks and demonstrate significant metric improvement over established pre-trained language models. We open-source TwHIN-BERT and our curated hashtag prediction and social engagement benchmark datasets to the research community.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Builds on 42 of its 62 references that are in the map. First posted on arXiv 15 Sep 2022.

Builds on

42 of 62 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  8. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  11. 2017Billion-scale similarity search with GPUscited by 2,477
  12. 2018Deep contextualized word representationscited by 2,285
30 more
  1. 2016node2vec: Scalable Feature Learning for Networkscited by 1,913
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  3. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  4. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  5. 2014DeepWalk: Online Learning of Social Representationscited by 1,543
  6. 2018Graph Convolutional Neural Networks for Web-Scale Recommender Systemscited by 1,133
  7. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  8. 2016Complex Embeddings for Simple Link Predictioncited by 897
  9. 2015LINE: Large-scale Information Network Embeddingcited by 582
  10. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 525
  11. 2020BERTweet: A pre-trained language model for English Tweetscited by 346
  12. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  13. 2019K-BERT: Enabling Language Representation with Knowledge Graphcited by 303
  14. 2019SemEval-2017 Task 4: Sentiment Analysis in Twittercited by 209
  15. 2022LinkBERT: Pretraining Language Models with Document Linkscited by 209
  16. 2022TimeLMs: Diachronic Language Models from Twittercited by 129
  17. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  18. 2019PyTorch-BigGraph: A Large-scale Graph Embedding Systemcited by 118
  19. 2014word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding methodcited by 109
  20. 2015A Survey of Heterogeneous Information Network Analysiscited by 109
  21. 2015PTE: Predictive Text Embedding through Large-scale Heterogeneous Text Networkscited by 96
  22. 2020SemEval-2020 Task 9: Overview of Sentiment Analysis of Code-Mixed Tweetscited by 62
  23. 2014Can Cascades be Predicted?cited by 31
  24. 2021Language Models are Few-Shot Butlerscited by 27
  25. 2022TwHIN: Embedding the Twitter Heterogeneous Information Network for Personalized Recommendationcited by 25
  26. 2021OAG-BERT: Towards A Unified Backbone Language Model For Academic Knowledge Servicescited by 20
  27. 2016Task-Guided and Path-Augmented Heterogeneous Network Embedding for Author Identificationcited by 18
  28. 2020Inf-VAE: A Variational Autoencoder Framework to Integrate Homophily and Influence in Diffusion Predictioncited by 10
  29. 2020ASAD: A Twitter-based Benchmark Arabic Sentiment Analysis Datasetcited by 7
  30. 2022kNN-Embed: Locally Smoothed Embedding Mixtures For Multi-interest Candidate Retrievalcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 17; 2024: 11; 2025: 7; 2026: 9.

44 papers.

Sort:

By area:

  1. 2023Large Language Models on Graphs: A Comprehensive Surveycited by 190
  2. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  3. 2023How Can Recommender Systems Benefit from Large Language Models: A Surveycited by 141
  4. 2024OASIS: Open Agent Social Interaction Simulations with One Million Agentscited by 127
  5. 2023When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunitiescited by 89
  6. 2023SemEval-2023 Task 10: Explainable Detection of Online Sexismcited by 48
  7. 2023Exploring Cross-Cultural Differences in English Hate Speech Annotations: From Dataset Construction to Analysiscited by 44
  8. 2023SemEval-2023 Task 12: Sentiment Analysis for African Languages (AfriSenti-SemEval)cited by 41
  9. 2024LLM-PBE: Assessing Data Privacy in Large Language Modelscited by 38
  10. 2023RAH! RecSys-Assistant-Human: A Human-Centered Recommendation Framework with LLM Agentscited by 30
  11. 2024Enhancing Recommendation Diversity by Re-ranking with Large Language Modelscited by 19
  12. 2023ViSoBERT: A Pre-Trained Language Model for Vietnamese Social Media Text Processingcited by 11
32 more
  1. 2024Does DetectGPT Fully Utilize Perturbation? Bridging Selective Perturbation to Fine-tuned Contrastive Learning Detector would be Bettercited by 11
  2. 2024Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Modelscited by 9
  3. 2025Training and Evaluating with Human Label Variation: An Empirical Studycited by 8
  4. 2023Balanced and Explainable Social Media Analysis for Public Health with Large Language Modelscited by 6
  5. 2025Don't Trust Generative Agents to Mimic Communication on Social Networks Unless You Benchmarked their Empirical Realismcited by 6
  6. 2024Relational Prompt-based Pre-trained Language Models for Social Event Detectioncited by 5
  7. 2024RuBia: A Russian Language Bias Detection Datasetcited by 4
  8. 2024Code-mixed Sentiment and Hate-speech Predictioncited by 4
  9. 2023Federated Learning Based Multilingual Emoji Prediction In Clean and Attack Scenarioscited by 3
  10. 2024Large Language Models Make Sample-Efficient Recommender Systemscited by 3
  11. 2023Multimodal Detection of Bots on X (Twitter) using Transformerscited by 2
  12. 2025Explaining Matters: Leveraging Definitions and Semantic Expansion for Sexism Detectioncited by 2
  13. 2025Enhancing Rumor Detection Methods with Propagation Structure Infused Language Modelcited by 2
  14. 2026PVminer: A Domain-Specific Tool to Detect the Patient Voice in Patient Generated Datacited by 2
  15. 2024Milestones in Bengali Sentiment Analysis leveraging Transformer-models: Fundamentals, Challenges and Future Directionscited by 1
  16. 2024Efficient Annotator Reliability Assessment and Sample Weighting for Knowledge-Based Misinformation Detection on Social Mediacited by 1
  17. 2026Learning Hierarchical Knowledge in Text-Rich Networks with Taxonomy-Informed Representation Learningcited by 1
  18. 2023RoBERTweet: A BERT Language Model for Romanian Tweetscited by 0
  19. 2023Politeness Stereotypes and Attack Vectors: Gender Stereotypes in Japanese and Korean Language Modelscited by 0
  20. 2023UPB at IberLEF-2023 AuTexTification: Detection of Machine-Generated Text using Transformer Ensemblescited by 0
  21. 2023HICL: Hashtag-Driven In-Context Learning for Social Media Natural Language Understandingcited by 0
  22. 2023URL-BERT: Training Webpage Representations via Social Media Engagementscited by 0
  23. 2025Full-Stack Optimized Large Language Models for Lifelong Sequential Behavior Comprehension in Recommendationcited by 0
  24. 2025Contrastive Learning Using Graph Embeddings for Domain Adaptation of Language Models in the Process Industrycited by 0
  25. 2025Seeing Hate Differently: Hate Subspace Modeling for Culture-Aware Hate Speech Detectioncited by 0
  26. 2026Recommender system in X inadvertently profiles ideological positions of userscited by 0
  27. 2026Assessing Capabilities of Large Language Models in Social Media Analytics: A Multi-task Questcited by 0
  28. 2026Pre-trained LLMs Meet Sequential Recommenders: Efficient User-Centric Knowledge Distillationcited by 0
  29. 2026The textitSilicon Society Cookbook: Design Space of LLM-based Social Simulationscited by 0
  30. 2026Can We Steer the Black-Box? Towards Controllability-Centric Evaluation of Recommender Systems with Collaborative Agentscited by 0
  31. 2026Against Political Polarization: A Unified Framework for Tracing Evolving Political Ideologies on Social Mediacited by 0
  32. 2026The Like Trap: Multi-Stage Poisoning against Agents in Similarity-based Recommendation Systemscited by 0