# ProgressGym: Alignment with a Millennium of Moral Progress

arXiv: https://arxiv.org/abs/2406.20087 · HTML: https://mlsanity.com/paper/2406.20087/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 125 references that are in the map.
- First posted on arXiv 28 Jun 2024.

## Builds on (most cited first, 30 of 52)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Aligning AI With Shared Human Values](https://mlsanity.com/paper/2008.02275/) (2020)
- [What learning algorithm is in-context learning? Investigations with linear models](https://mlsanity.com/paper/2211.15661/) (2022)
- [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://mlsanity.com/paper/2108.08877/) (2021)
- [Artificial Intelligence, Values and Alignment](https://mlsanity.com/paper/2001.09768/) (2020)
- [Unsolved Problems in ML Safety](https://mlsanity.com/paper/2109.13916/) (2021)
- [Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning](https://mlsanity.com/paper/2011.01403/) (2020)
- [AI Alignment: A Comprehensive Survey](https://mlsanity.com/paper/2310.19852/) (2023)
- [TidyBot: Personalized Robot Assistance with Large Language Models](https://mlsanity.com/paper/2305.05658/) (2023)
- [Evaluating the Moral Beliefs Encoded in LLMs](https://mlsanity.com/paper/2307.14324/) (2023)
- [Foundational Challenges in Assuring Alignment and Safety of Large Language Models](https://mlsanity.com/paper/2404.09932/) (2024)
- [Fine-tuning language models to find agreement among humans with diverse preferences](https://mlsanity.com/paper/2211.15006/) (2022)
- [Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark](https://mlsanity.com/paper/2304.03279/) (2023)
- [The Capacity for Moral Self-Correction in Large Language Models](https://mlsanity.com/paper/2302.07459/) (2023)
- [Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigations](https://mlsanity.com/paper/2301.04246/) (2023)
- [Co-Writing with Opinionated Language Models Affects Users' Views](https://mlsanity.com/paper/2302.00560/) (2023)

## Built on by (most cited first, 9 of 9)

- [Amulet: ReAlignment During Test Time for Personalized Preference Adaptation of LLMs](https://mlsanity.com/paper/2502.19148/) (2025)
- [The Lock-in Hypothesis: Stagnation by Algorithm](https://mlsanity.com/paper/2506.06166/) (2025)
- [CLASH: Evaluating Language Models on Judging High-Stakes Dilemmas from Multiple Perspectives](https://mlsanity.com/paper/2504.10823/) (2025)
- [Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectives](https://mlsanity.com/paper/2506.09656/) (2025)
- [Positive Alignment: Artificial Intelligence for Human Flourishing](https://mlsanity.com/paper/2605.10310/) (2026)
- [Super Co-alignment of Human and AI for Sustainable Symbiotic Society](https://mlsanity.com/paper/2504.17404/) (2025)
- [A Game-Theoretic Negotiation Framework for Cross-Cultural Consensus in LLMs](https://mlsanity.com/paper/2506.13245/) (2025)
- [Adaptive Pluralistic Alignment: A pipeline for dynamic artificial democracy](https://mlsanity.com/paper/2605.01642/) (2026)
- [Quantifying Risk Propensities of Large Language Models: Ethical Focus and Bias Detection through Role-Play](https://mlsanity.com/paper/2411.08884/) (2024)

## Abstract (arXiv)

Frontier AI systems, including large language models (LLMs), hold increasing influence over the epistemology of human users. Such influence can reinforce prevailing societal values, potentially contributing to the lock-in of misguided moral beliefs and, consequently, the perpetuation of problematic moral practices on a broad scale. We introduce progress alignment as a technical solution to mitigate this imminent risk. Progress alignment algorithms learn to emulate the mechanics of human moral progress, thereby addressing the susceptibility of existing alignment methods to contemporary moral blindspots. To empower research in progress alignment, we introduce ProgressGym, an experimental framework allowing the learning of moral progress mechanics from history, in order to facilitate future progress in real-world moral decisions. Leveraging 9 centuries of historical text and 18 historical LLMs, ProgressGym enables codification of real-world progress alignment challenges into concrete benchmarks. Specifically, we introduce three core challenges: tracking evolving values (PG-Follow), preemptively anticipating moral progress (PG-Predict), and regulating the feedback loop between human and AI value shifts (PG-Coevolve). Alignment methods without a temporal dimension are inapplicable to these tasks. In response, we present lifelong and extrapolative algorithms as baseline methods of progress alignment, and build an open leaderboard soliciting novel algorithms and challenges. The framework and the leaderboard are available at https://github.com/PKU-Alignment/ProgressGym and https://huggingface.co/spaces/PKU-Alignment/ProgressGym-LeaderBoard respectively.