# Word Overuse and Alignment in Large Language Models: The Influence of Learning from Human Feedback

arXiv: https://arxiv.org/abs/2508.01930 · HTML: https://mlsanity.com/paper/2508.01930/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 24 of its 67 references that are in the map.
- First posted on arXiv 3 Aug 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 24 of 24)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature](https://mlsanity.com/paper/2301.11305/) (2023)
- [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://mlsanity.com/paper/2306.16388/) (2023)
- [Explainability for Large Language Models: A Survey](https://mlsanity.com/paper/2309.01029/) (2023)
- [Mapping the Increasing Use of LLMs in Scientific Papers](https://mlsanity.com/paper/2404.01268/) (2024)
- [On the Possibilities of AI-Generated Text Detection](https://mlsanity.com/paper/2304.04736/) (2023)
- [Style Over Substance: Evaluation Biases for Large Language Models](https://mlsanity.com/paper/2307.03025/) (2023)
- [From Lists to Emojis: How Format Bias Affects Model Alignment](https://mlsanity.com/paper/2409.11704/) (2024)
- [Empirical evidence of Large Language Model's influence on human spoken communication](https://mlsanity.com/paper/2409.01754/) (2024)
- [ChatGPT "contamination": estimating the prevalence of LLMs in the scholarly literature](https://mlsanity.com/paper/2403.16887/) (2024)
- [XAI meets LLMs: A Survey of the Relation between Explainable AI and Large Language Models](https://mlsanity.com/paper/2407.15248/) (2024)
- [Whose Emotions and Moral Sentiments Do Language Models Reflect?](https://mlsanity.com/paper/2402.11114/) (2024)
- [The Impact of Large Language Models in Academia: from Writing to Speaking](https://mlsanity.com/paper/2409.13686/) (2024)
- [Quantifying the Invisible Labor in Crowd Work](https://mlsanity.com/paper/2110.00169/) (2021)
- [The Role of AI in Peer Support for Young People: A Study of Preferences for Human- and AI-Generated Responses](https://mlsanity.com/paper/2405.02711/) (2024)
- [Towards the relationship between AIGC in manuscript writing and author profiles: evidence from preprints in LLMs](https://mlsanity.com/paper/2404.15799/) (2024)
- [Comparing LLM-generated and human-authored news text using formal syntactic theory](https://mlsanity.com/paper/2506.01407/) (2025)
- [Why Does ChatGPT "Delve" So Much? Exploring the Sources of Lexical Overrepresentation in Large Language Models](https://mlsanity.com/paper/2412.11385/) (2024)

## Built on by (most cited first, 2 of 2)

- [Isolating LLM Lexical Bias: A Curation-Free Triangulated Metric for Preference-Stage Learning](https://mlsanity.com/paper/2606.00334/) (2026)
- [Where Should Language Sit in a Multimodal Model? Lessons from What Language Does to Human Perception and Cognition](https://mlsanity.com/paper/2609.07474/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are known to overuse certain terms like "delve" and "intricate." The exact reasons for these lexical choices, however, have been unclear. Using Meta's Llama model, this study investigates the contribution of Learning from Human Feedback (LHF), under which we subsume Reinforcement Learning from Human Feedback and Direct Preference Optimization. We present a straightforward procedure for detecting the lexical preferences of LLMs that are potentially LHF-induced. Next, we more conclusively link LHF to lexical overuse by experimentally emulating the LHF procedure and demonstrating that participants systematically prefer text variants that include certain words. This lexical overuse can be seen as a sort of misalignment, though our study highlights the potential divergence between the lexical expectations of different populations -- namely LHF workers versus LLM users. Our work contributes to the growing body of research on explainable artificial intelligence and emphasizes the importance of both data and procedural transparency in alignment research.