SMILE: Evaluation and Domain Adaptation for Social Media Language Understanding
Not on the map. Why?
Abstract (arXiv)
We study the ability of transformer-based language models (LMs) to understand social media language. Social media (SM) language is distinct from standard written language, yet existing benchmarks fall short of capturing LM performance in this socially, economically, and politically important domain. We quantify the degree to which social media language differs from conventional language and conclude that the difference is significant both in terms of token distribution and rate of linguistic shift. Next, we introduce a new benchmark for Social MedIa Language Evaluation (SMILE) that covers four SM platforms and eleven tasks. Finally, we show that learning a tokenizer and pretraining on a mix of social media and conventional language yields an LM that outperforms the best similar-sized alternative by 4.2 points on the overall SMILE score.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 42 of its 61 references that are in the map. First posted on arXiv 30 Jun 2023.
Builds on
42 of 61 references.
References · 42
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
- 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
- 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
30 more
- 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
- 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
- 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
- 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
- 2020Rethinking Attention with Performerscited by 1,601
- 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
- 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
- 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
- 2020Efficient Transformers: A Surveycited by 790
- 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 629
- 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 538
- 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
- 2020GoEmotions: A Dataset of Fine-Grained Emotionscited by 486
- 2020TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classificationcited by 417
- 2019Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classificationcited by 404
- 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 393
- 2020BERTweet: A pre-trained language model for English Tweetscited by 346
- 2022GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Recordscited by 238
- 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
- 2021MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcarecited by 161
- 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 160
- 2021XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyondcited by 114
- 2018Abstractive Summarization of Reddit Posts with Multi-level Memory Networkscited by 102
- 2020Minimizing FLOPs to Learn Efficient Sparse Representationscited by 46
- 2021Dynamic Language Models for Continuously Evolving Contentcited by 27
- 2019Partially Linear Additive Gaussian Graphical Modelscited by 6
- 2020Temporal Poisson Square Root Graphical Modelscited by 6
- 2018Joint Nonparametric Precision Matrix Estimation with Confoundingcited by 4
- 2020Stochastic Learning for Sparse Discrete Markov Random Fields with Controlled Gradient Approximation Errorcited by 4
- 2017An Efficient Pseudo-likelihood Method for Sparse Binary Pairwise Markov Network Estimationcited by 3
Built on by
Citing papers per year: 2024: 1.
1 paper.