# Towards Scalable Automated Alignment of LLMs: A Survey

arXiv: https://arxiv.org/abs/2406.01252 · HTML: https://mlsanity.com/paper/2406.01252/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jun 2024.

## Built on by (most cited first, 30 of 35)

- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://mlsanity.com/paper/2409.12122/) (2024)
- [ProcessBench: Identifying Process Errors in Mathematical Reasoning](https://mlsanity.com/paper/2412.06559/) (2024)
- [Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models](https://mlsanity.com/paper/2406.13542/) (2024)
- [Aligning LLMs with Individual Preferences via Interaction](https://mlsanity.com/paper/2410.03642/) (2024)
- [Critic-CoT: Boosting the reasoning abilities of large language model via Chain-of-thoughts Critic](https://mlsanity.com/paper/2408.16326/) (2024)
- [Language Models can Self-Lengthen to Generate Long Texts](https://mlsanity.com/paper/2410.23933/) (2024)
- [SPaR: Self-Play with Tree-Search Refinement to Improve Instruction-Following in Large Language Models](https://mlsanity.com/paper/2412.11605/) (2024)
- [AURORA:Automated Training Framework of Universal Process Reward Models via Ensemble Prompting and Reverse Verification](https://mlsanity.com/paper/2502.11520/) (2025)
- [MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimization](https://mlsanity.com/paper/2410.07672/) (2024)
- [MPO: Multilingual Safety Alignment via Reward Gap Optimization](https://mlsanity.com/paper/2505.16869/) (2025)
- [Search, Verify and Feedback: Towards Next Generation Post-training Paradigm of Foundation Models via Verifier Engineering](https://mlsanity.com/paper/2411.11504/) (2024)
- [Debiasing Multimodal Large Language Models via Noise-Aware Preference Optimization](https://mlsanity.com/paper/2503.17928/) (2025)
- [Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Models](https://mlsanity.com/paper/2411.08733/) (2024)
- [From Generalist to Specialist: A Survey of Large Language Models for Chemistry](https://mlsanity.com/paper/2412.19994/) (2024)
- [Toward General Instruction-Following Alignment for Retrieval-Augmented Generation](https://mlsanity.com/paper/2410.09584/) (2024)
- [How Do Your Code LLMs Perform? Empowering Code Instruction Tuning with High-Quality Data](https://mlsanity.com/paper/2409.03810/) (2024)
- [Position: Towards a Responsible LLM-empowered Multi-Agent Systems](https://mlsanity.com/paper/2502.01714/) (2025)
- [Lens: Rethinking Multilingual Enhancement for Large Language Models](https://mlsanity.com/paper/2410.04407/) (2024)
- [Adaptive Alignment: Dynamic Preference Adjustments via Multi-Objective Reinforcement Learning for Pluralistic AI](https://mlsanity.com/paper/2410.23630/) (2024)
- [Self-Steering Optimization: Autonomous Preference Optimization for Large Language Models](https://mlsanity.com/paper/2410.17131/) (2024)
- [Evaluating the Robustness of Large Language Model Safety Guardrails Against Adversarial Attacks](https://mlsanity.com/paper/2511.22047/) (2025)
- [Representational Alignment with Chemical Induced Fit for Molecular Relational Learning](https://mlsanity.com/paper/2502.07027/) (2025)
- [From Anger to Joy: How Nationality Personas Shape Emotion Attribution in Large Language Models](https://mlsanity.com/paper/2506.02431/) (2025)
- [Contrastive Weak-to-strong Generalization](https://mlsanity.com/paper/2510.07884/) (2025)
- [Aligning Deep Implicit Preferences by Learning to Reason Defensively](https://mlsanity.com/paper/2510.11194/) (2025)
- [Transferable Post-training via Inverse Value Learning](https://mlsanity.com/paper/2410.21027/) (2024)
- [Superalignment with Dynamic Human Values](https://mlsanity.com/paper/2503.13621/) (2025)
- [An Uncertainty-Driven Adaptive Self-Alignment Framework for Large Language Models](https://mlsanity.com/paper/2507.17477/) (2025)

## Abstract (arXiv)

Alignment is the most critical step in building large language models (LLMs) that meet human needs. With the rapid development of LLMs gradually surpassing human capabilities, traditional alignment methods based on human-annotation are increasingly unable to meet the scalability demands. Therefore, there is an urgent need to explore new sources of automated alignment signals and technical approaches. In this paper, we systematically review the recently emerging methods of automated alignment, attempting to explore how to achieve effective, scalable, automated alignment once the capabilities of LLMs exceed those of humans. Specifically, we categorize existing automated alignment methods into 4 major categories based on the sources of alignment signals and discuss the current status and potential development of each category. Additionally, we explore the underlying mechanisms that enable automated alignment and discuss the essential factors that make automated alignment technologies feasible and effective from the fundamental role of alignment.