mlsanityData to 14 Oct 2026

Methodological reflections for AI alignment research using human feedback

arXiv 2301.06859 ↗First posted 22 Dec 2022cs.HC cs.AI cs.CLcited by 5

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

The field of artificial intelligence (AI) alignment aims to investigate whether AI technologies align with human interests and values and function in a safe and ethical manner. AI alignment is particularly relevant for large language models (LLMs), which have the potential to exhibit unintended behavior due to their ability to learn and adapt in ways that are difficult to predict. In this paper, we discuss methodological challenges for the alignment problem specifically in the context of LLMs trained to summarize texts. In particular, we focus on methods for collecting reliable human feedback on summaries to train a reward model which in turn improves the summarization model. We conclude by suggesting specific improvements in the experimental design of alignment studies for LLMs' summarization capabilities.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 10 of its 20 references that are in the map. First posted on arXiv 22 Dec 2022.

Builds on

10 of 20 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020Learning to summarize from human feedback2 Sep

Other references · 9

  1. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  2. 2016Concrete Problems in AI Safetycited by 1,642
  3. 2021Ethical and social risks of harm from Language Modelscited by 966
  4. 2020Artificial Intelligence, Values and Alignmentcited by 448
  5. 2022Self-critiquing models for assisting human evaluatorscited by 380
  6. 2020TLDR: Extreme Summarization of Scientific Documentscited by 136
  7. 2020Multi-XScience: A Large-scale Dataset for Extreme Multi-document Summarization of Scientific Articlescited by 97
  8. 2020What Have We Achieved on Text Summarization?cited by 65
  9. 2021Fairness for Whom? Understanding the Reader's Perception of Fairness in Text Summarizationcited by 8

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1; 2025: 3.

5 papers.

  1. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 264
  2. 2024Risks of AI Scientists: Prioritizing Safeguarding Over Autonomycited by 49
  3. 2025Generative Artificial Intelligence: Evolving Technology, Growing Societal Impact, and Opportunities for Information Systems Researchcited by 4
  4. 2025On the Inevitability of Left-Leaning Political Bias in Aligned Language Modelscited by 4
  5. 2025Epitome: Pioneering an Experimental Platform for AI-Social Science Integrationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Decoding Alignment: A Critical Survey of LLM Development Initiatives through Value-setting and Data-centric Lens
  2. 2025Evaluating AI Alignment in LLMs: Output Analysis of Value Priorities Across 75 Models with Human Benchmarking
  3. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  4. 2023Aligning Large Language Models with Human: A Survey
  5. 2025Dynamic Alignment for Collective Agency: Toward a Scalable Self-Improving Framework for Open-Ended LLM Alignment
  6. 2023Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models
  7. 2023Large Language Model Alignment: A Survey
  8. 2025DxHF: Providing High-Quality Human Feedback for LLM Alignment via Interactive Decomposition