# Addressing the Ecological Fallacy in Larger LMs with Human Context

arXiv: https://arxiv.org/abs/2603.05928 · HTML: https://mlsanity.com/paper/2603.05928/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 34 references that are in the map.
- First posted on arXiv 6 Mar 2026.

## Builds on (most cited first, 19 of 19)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoders](https://mlsanity.com/paper/2403.03952/) (2024)
- [From Amateurs to Connoisseurs: Modeling the Evolution of User Expertise through Online Reviews](https://mlsanity.com/paper/1303.4402/) (2013)
- [Democratizing Large Language Models via Personalized Parameter-Efficient Fine-tuning](https://mlsanity.com/paper/2402.04401/) (2024)
- [Yelp Dataset Challenge: Review Rating Prediction](https://mlsanity.com/paper/1605.05362/) (2016)
- [Aligning Large Language Models with Human Preferences through Representation Engineering](https://mlsanity.com/paper/2312.15997/) (2023)
- [UserIdentifier: Implicit User Representations for Simple and Effective Personalized Sentiment Analysis](https://mlsanity.com/paper/2110.00135/) (2021)
- [SemEval-2013 Task 2: Sentiment Analysis in Twitter](https://mlsanity.com/paper/1912.06806/) (2019)
- [Empathetic Dialogue Generation via Sensitive Emotion Recognition and Sensible Knowledge Selection](https://mlsanity.com/paper/2210.11715/) (2022)
- [MeLT: Message-Level Transformer with Masked Document Representations as Pre-Training for Stance Detection](https://mlsanity.com/paper/2109.08113/) (2021)
- [Large Human Language Models: A Need and the Challenges](https://mlsanity.com/paper/2312.07751/) (2023)
- [Comparing Pre-trained Human Language Models: Is it Better with Human Context as Groups, Individual Traits, or Both?](https://mlsanity.com/paper/2401.12492/) (2024)
- [Evaluation of LLMs-based Hidden States as Author Representations for Psychological Human-Centered NLP Tasks](https://mlsanity.com/paper/2503.00124/) (2025)

## Abstract (arXiv)

Language model training and inference ignore a fundamental linguistic fact -- there is a dependence between multiple sequences of text written by the same person. Prior work has shown that addressing this form of textitecological fallacy can greatly improve the performance of multiple smaller (~124M) GPT-based models. In this work, we ask if addressing the ecological fallacy by modeling the author's language context with a specific LM task (called HuLM) can provide similar benefits for a larger-scale model, an 8B Llama model. To this end, we explore variants that process an author's language in the context of their other temporally ordered texts. We study the effect of pre-training with this author context using the HuLM objective, as well as using it during fine-tuning with author context (textitHuFT:Human-aware Fine-Tuning). Empirical comparisons show that addressing the ecological fallacy during fine-tuning alone using QLoRA improves the performance of the larger 8B model over standard fine-tuning. Additionally, QLoRA-based continued HuLM pre-training results in a human-aware model generalizable for improved performance over eight downstream tasks with linear task classifier training alone. These results indicate the utility and importance of modeling language in the context of its original generators, the authors.