# NTULM: Enriching Social Media Text Representations with Non-Textual Units

arXiv: https://arxiv.org/abs/2210.16586 · HTML: https://mlsanity.com/paper/2210.16586/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 30 references that are in the map.
- First posted on arXiv 29 Oct 2022.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification](https://mlsanity.com/paper/2010.12421/) (2020)
- [KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation](https://mlsanity.com/paper/1911.06136/) (2019)
- [K-BERT: Enabling Language Representation with Knowledge Graph](https://mlsanity.com/paper/1909.07606/) (2019)
- [LinkBERT: Pretraining Language Models with Document Links](https://mlsanity.com/paper/2203.15827/) (2022)
- [word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method](https://mlsanity.com/paper/1402.3722/) (2014)
- [TIMME: Twitter Ideology-detection via Multi-task Multi-relational Embedding](https://mlsanity.com/paper/2006.01321/) (2020)
- [Computational Sociolinguistics: A Survey](https://mlsanity.com/paper/1508.07544/) (2015)
- [TwHIN: Embedding the Twitter Heterogeneous Information Network for Personalized Recommendation](https://mlsanity.com/paper/2202.05387/) (2022)
- [TweetBERT: A Pretrained Language Representation Model for Twitter Text Analysis](https://mlsanity.com/paper/2010.11091/) (2020)
- [Unsupervised Belief Representation Learning with Information-Theoretic Variational Graph Auto-Encoders](https://mlsanity.com/paper/2110.00210/) (2021)
- [KI-BERT: Infusing Knowledge Context for Better Language and Domain Understanding](https://mlsanity.com/paper/2104.08145/) (2021)
- [Tweet2Vec: Learning Tweet Embeddings Using Character-level CNN-LSTM Encoder-Decoder](https://mlsanity.com/paper/1607.07514/) (2016)
- [LMSOC: An Approach for Socially Sensitive Pretraining](https://mlsanity.com/paper/2110.10319/) (2021)
- [Learning Stance Embeddings from Signed Social Graphs](https://mlsanity.com/paper/2201.11675/) (2022)
- [Multi-task Pairwise Neural Ranking for Hashtag Segmentation](https://mlsanity.com/paper/1906.00790/) (2019)
- [kNN-Embed: Locally Smoothed Embedding Mixtures For Multi-interest Candidate Retrieval](https://mlsanity.com/paper/2205.06205/) (2022)
- [TweeTime: A Minimally Supervised Method for Recognizing and Normalizing Time Expressions in Twitter](https://mlsanity.com/paper/1608.02904/) (2016)
- [CTM -- A Model for Large-Scale Multi-View Tweet Topic Classification](https://mlsanity.com/paper/2205.01603/) (2022)

## Built on by (most cited first, 2 of 2)

- [ConGraT: Self-Supervised Contrastive Pretraining for Joint Graph and Text Embeddings](https://mlsanity.com/paper/2305.14321/) (2023)
- [MAPLE: Metadata Conditioned LLM Pretraining for Locale-Aware Question Answering](https://mlsanity.com/paper/2601.15236/) (2026)

## Abstract (arXiv)

On social media, additional context is often present in the form of annotations and meta-data such as the post's author, mentions, Hashtags, and hyperlinks. We refer to these annotations as Non-Textual Units (NTUs). We posit that NTUs provide social context beyond their textual semantics and leveraging these units can enrich social media text representations. In this work we construct an NTU-centric social heterogeneous network to co-embed NTUs. We then principally integrate these NTU embeddings into a large pretrained language model by fine-tuning with these additional units. This adds context to noisy short-text social media. Experiments show that utilizing NTU-augmented text representations significantly outperforms existing text-only baselines by 2-5% relative points on many downstream tasks highlighting the importance of context to social media NLP. We also highlight that including NTU context into the initial layers of language model alongside text is better than using it after the text embedding is generated. Our work leads to the generation of holistic general purpose social media content embedding.