# Bias Amplification in Language Model Evolution: An Iterated Learning Perspective

arXiv: https://arxiv.org/abs/2404.04286 · HTML: https://mlsanity.com/paper/2404.04286/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 53 references that are in the map.
- First posted on arXiv 4 Apr 2024.

## Builds on (most cited first, 26 of 26)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment](https://mlsanity.com/paper/2304.06767/) (2023)
- [Self-Rewarding Language Models](https://mlsanity.com/paper/2401.10020/) (2024)
- [Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models](https://mlsanity.com/paper/2401.01335/) (2024)
- [Reinforced Self-Training (ReST) for Language Modeling](https://mlsanity.com/paper/2308.08998/) (2023)
- [Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision](https://mlsanity.com/paper/2312.09390/) (2023)
- [Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint](https://mlsanity.com/paper/2312.11456/) (2023)
- [Direct Language Model Alignment from Online AI Feedback](https://mlsanity.com/paper/2402.04792/) (2024)
- [Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks](https://mlsanity.com/paper/2307.02477/) (2023)
- [Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve](https://mlsanity.com/paper/2309.13638/) (2023)
- [Self-Distillation Amplifies Regularization in Hilbert Space](https://mlsanity.com/paper/2002.05715/) (2020)
- [Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement](https://mlsanity.com/paper/2310.08559/) (2023)
- [Compositional Languages Emerge in a Neural Iterated Learning Model](https://mlsanity.com/paper/2002.01365/) (2020)
- [Countering Language Drift with Seeded Iterated Learning](https://mlsanity.com/paper/2003.12694/) (2020)
- [ACRE: Abstract Causal REasoning Beyond Covariation](https://mlsanity.com/paper/2103.14232/) (2021)
- [The Emergence of Compositional Languages for Numeric Concepts Through Iterated Learning in Neural Agents](https://mlsanity.com/paper/1910.05291/) (2019)
- [Bayes in the age of intelligent machines](https://mlsanity.com/paper/2311.10206/) (2023)
- [Iterated learning for emergent systematicity in VQA](https://mlsanity.com/paper/2105.01119/) (2021)
- [Iterated Learning Improves Compositionality in Large Vision-Language Models](https://mlsanity.com/paper/2404.02145/) (2024)
- [The cognitive roots of regularization in language](https://mlsanity.com/paper/1703.03442/) (2017)

## Built on by (most cited first, 17 of 17)

- [Learning Dynamics of LLM Finetuning](https://mlsanity.com/paper/2407.10490/) (2024)
- [MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimization](https://mlsanity.com/paper/2410.07672/) (2024)
- [A Survey on Collaborative Mechanisms Between Large and Small Language Models](https://mlsanity.com/paper/2505.07460/) (2025)
- [Exploring Silicon-Based Societies: An Early Study of the Moltbook Agent Community](https://mlsanity.com/paper/2602.02613/) (2026)
- [The Lock-in Hypothesis: Stagnation by Algorithm](https://mlsanity.com/paper/2506.06166/) (2025)
- [Shaping Shared Languages: Human and Large Language Models' Inductive Biases in Emergent Communication](https://mlsanity.com/paper/2503.04395/) (2025)
- [Iterative Label Refinement Matters More than Preference Optimization under Weak Supervision](https://mlsanity.com/paper/2501.07886/) (2025)
- [Review of Hallucination Understanding in Large Language and Vision Models](https://mlsanity.com/paper/2510.00034/) (2025)
- [Losing our Tail, Again: (Un)Natural Selection & Multilingual LLMs](https://mlsanity.com/paper/2507.03933/) (2025)
- [Evolution and compression in LLMs: On the emergence of human-aligned categorization](https://mlsanity.com/paper/2509.08093/) (2025)
- [Bias in the Tails: How Name-conditioned Evaluative Framing in Resume Summaries Destabilizes LLM-based Hiring](https://mlsanity.com/paper/2604.19984/) (2026)
- [Model Collapse as Cultural Evolution](https://mlsanity.com/paper/2605.23054/) (2026)
- [GlossoGen: Emergent Language in Complex Multi-Agent LLM Interactions](https://mlsanity.com/paper/2609.01491/) (2026)
- [Iterative Finetuning is Mostly Idempotent](https://mlsanity.com/paper/2605.01130/) (2026)
- [Team-Based Self-Play With Dual Adaptive Weighting for Fine-Tuning LLMs](https://mlsanity.com/paper/2605.09922/) (2026)
- [Distill to Detect: Exposing Stealth Biases in LLMs through Cartridge Distillation](https://mlsanity.com/paper/2607.01208/) (2026)
- [Learning Dynamics of Continual Learning: A Unified View of Data Attribution, Forgetting, and Plasticity Loss](https://mlsanity.com/paper/2609.33620/) (2026)

## Abstract (arXiv)

With the widespread adoption of Large Language Models (LLMs), the prevalence of iterative interactions among these models is anticipated to increase. Notably, recent advancements in multi-round self-improving methods allow LLMs to generate new examples for training subsequent models. At the same time, multi-agent LLM systems, involving automated interactions among agents, are also increasing in prominence. Thus, in both short and long terms, LLMs may actively engage in an evolutionary process. We draw parallels between the behavior of LLMs and the evolution of human culture, as the latter has been extensively studied by cognitive scientists for decades. Our approach involves leveraging Iterated Learning (IL), a Bayesian framework that elucidates how subtle biases are magnified during human cultural evolution, to explain some behaviors of LLMs. This paper outlines key characteristics of agents' behavior in the Bayesian-IL framework, including predictions that are supported by experimental verification with various LLMs. This theoretical framework could help to more effectively predict and guide the evolution of LLMs in desired directions.