# Countering Language Drift with Seeded Iterated Learning

arXiv: https://arxiv.org/abs/2003.12694 · HTML: https://mlsanity.com/paper/2003.12694/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Mar 2020.

## Built on by (most cited first, 30 of 57)

- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022)
- [A General Theoretical Paradigm to Understand Learning from Human Preferences](https://mlsanity.com/paper/2310.12036/) (2023)
- [Reinforced Self-Training (ReST) for Language Modeling](https://mlsanity.com/paper/2308.08998/) (2023)
- [SINE: SINgle Image Editing with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2212.04489/) (2022)
- [Cones: Concept Neurons in Diffusion Models for Customized Generation](https://mlsanity.com/paper/2303.05125/) (2023)
- [WARM: On the Benefits of Weight Averaged Reward Models](https://mlsanity.com/paper/2401.12187/) (2024)
- [A Recipe for Watermarking Diffusion Models](https://mlsanity.com/paper/2303.10137/) (2023)
- [ITI-GEN: Inclusive Text-to-Image Generation](https://mlsanity.com/paper/2309.05569/) (2023)
- [DreamMatcher: Appearance Matching Self-Attention for Semantically-Consistent Text-to-Image Personalization](https://mlsanity.com/paper/2402.09812/) (2024)
- [Customizing 360-Degree Panoramas through Text-to-Image Diffusion Models](https://mlsanity.com/paper/2310.18840/) (2023)
- [WARP: On the Benefits of Weight Averaged Rewarded Policies](https://mlsanity.com/paper/2406.16768/) (2024)
- [Fortuitous Forgetting in Connectionist Networks](https://mlsanity.com/paper/2202.00155/) (2022)
- [Language Model Alignment with Elastic Reset](https://mlsanity.com/paper/2312.07551/) (2023)
- [All but One: Surgical Concept Erasing with Model Preservation in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2312.12807/) (2023)
- [Attention Calibration for Disentangled Text-to-Image Personalization](https://mlsanity.com/paper/2403.18551/) (2024)
- [AvatarStudio: Text-driven Editing of 3D Dynamic Human Head Avatars](https://mlsanity.com/paper/2306.00547/) (2023)
- [SleeperMark: Towards Robust Watermark against Fine-Tuning Text-to-image Diffusion Models](https://mlsanity.com/paper/2412.04852/) (2024)
- [ViCo: Plug-and-play Visual Condition for Personalized Text-to-image Generation](https://mlsanity.com/paper/2306.00971/) (2023)
- [Iterated learning for emergent systematicity in VQA](https://mlsanity.com/paper/2105.01119/) (2021)
- [Countering Reward Over-optimization in LLM with Demonstration-Guided Reinforcement Learning](https://mlsanity.com/paper/2404.19409/) (2024)
- [Multi-label Iterated Learning for Image Classification with Label Ambiguity](https://mlsanity.com/paper/2111.12172/) (2021)
- [Eliciting the Priors of Large Language Models using Iterated In-Context Learning](https://mlsanity.com/paper/2406.01860/) (2024)
- [NewMove: Customizing text-to-video models with novel motions](https://mlsanity.com/paper/2312.04966/) (2023)
- [Bias Amplification in Language Model Evolution: An Iterated Learning Perspective](https://mlsanity.com/paper/2404.04286/) (2024)
- [A Review of the Applications of Deep Learning-Based Emergent Communication](https://mlsanity.com/paper/2407.03302/) (2024)
- [Improving Compositional Generalization Using Iterated Learning and Simplicial Embeddings](https://mlsanity.com/paper/2310.18777/) (2023)
- [Generate Anything Anywhere in Any Scene](https://mlsanity.com/paper/2306.17154/) (2023)
- [The Emergence of the Shape Bias Results from Communicative Efficiency](https://mlsanity.com/paper/2109.06232/) (2021)
- [Towards More Human-like AI Communication: A Review of Emergent Communication Research](https://mlsanity.com/paper/2308.02541/) (2023)

## Abstract (arXiv)

Pretraining on human corpus and then finetuning in a simulator has become a standard pipeline for training a goal-oriented dialogue agent. Nevertheless, as soon as the agents are finetuned to maximize task completion, they suffer from the so-called language drift phenomenon: they slowly lose syntactic and semantic properties of language as they only focus on solving the task. In this paper, we propose a generic approach to counter language drift called Seeded iterated learning (SIL). We periodically refine a pretrained student agent by imitating data sampled from a newly generated teacher agent. At each time step, the teacher is created by copying the student agent, before being finetuned to maximize task completion. SIL does not require external syntactic constraint nor semantic knowledge, making it a valuable task-agnostic finetuning protocol. We evaluate SIL in a toy-setting Lewis Game, and then scale it up to the translation game with natural language. In both settings, SIL helps counter language drift as well as it improves the task completion compared to baselines.