mlsanityData to 14 Oct 2026

Human Language Modeling

arXiv 2205.05128 ↗First posted 10 May 2022cs.CL cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Natural language is generated by people, yet traditional language modeling views words or documents as if generated independently. Here, we propose human language modeling (HuLM), a hierarchical extension to the language modeling problem whereby a human-level exists to connect sequences of documents (e.g. social media messages) and capture the notion that human language is moderated by changing human states. We introduce, HaRT, a large-scale transformer model for the HuLM task, pre-trained on approximately 100,000 social media users, and demonstrate its effectiveness in terms of both language modeling (perplexity) for social media and fine-tuning for 4 downstream tasks spanning document- and user-levels: stance detection, sentiment classification, age estimation, and personality assessment. Results on all tasks meet or surpass the current state-of-the-art.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 23 of its 49 references that are in the map. First posted on arXiv 10 May 2022.

Builds on

23 of 49 references.

References · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2020Longformer: The Long-Document Transformercited by 3,413
  4. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  5. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,501
  6. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  7. 2019Language Models as Knowledge Bases?cited by 1,936
  8. 2019How Can We Know What Language Models Know?cited by 959
  9. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  10. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 799
  11. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 762
  12. 2020BERTweet: A pre-trained language model for English Tweetscited by 346
11 more
  1. 2019Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overviewcited by 137
  2. 2021UserIdentifier: Implicit User Representations for Simple and Effective Personalized Sentiment Analysiscited by 28
  3. 2018Personalized Language Model for Query Auto-Completioncited by 25
  4. 2020Compositional Demographic Word Embeddingscited by 20
  5. 2020Exploring the Value of Personalized Word Embeddingscited by 17
  6. 2021Empirical Evaluation of Pre-trained Transformers for Human-Level NLP: The Role of Sample Size and Dimensionalitycited by 15
  7. 2021MeLT: Message-Level Transformer with Masked Document Representations as Pre-Training for Stance Detectioncited by 11
  8. 2018The Remarkable Benefit of User-Level Aggregation for Lexical-based Population-Level Predictionscited by 10
  9. 2019Learning Dynamic Author Representations with Temporal Language Modelscited by 5
  10. 2020Adding Recurrence to Pretrained Transformers for Improved Efficiency and Context Sizecited by 5
  11. 2017Personalized word representations Carrying Personalized Semantics Learned from Social Network Postscited by 2

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 2; 2025: 1.

5 papers.

  1. 2023LaMP: When Large Language Models Meet Personalizationcited by 437
  2. 2024User Embedding Model for Personalized Language Promptingcited by 28
  3. 2024Towards Unified Multi-Modal Personalization: Large Vision-Language Models for Generative Recommendation and Beyondcited by 25
  4. 2023Exploring the Landscape of Natural Language Processing Researchcited by 7
  5. 2025WhiSPA: Semantically and Psychologically Aligned Whisper with Self-Supervised Contrastive and Student-Teacher Learningcited by 3