mlsanityData to 7 Oct 2026

Value Alignment from Unstructured Text

arXiv 2408.10392 ↗First posted 19 Aug 2024cs.CL cs.LGcited by 4

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Aligning large language models (LLMs) to value systems has emerged as a significant area of research within the fields of AI and NLP. Currently, this alignment process relies on the availability of high-quality supervised and preference data, which can be both time-consuming and expensive to curate or annotate. In this paper, we introduce a systematic end-to-end methodology for aligning LLMs to the implicit and explicit values represented in unstructured text data. Our proposed approach leverages the use of scalable synthetic data generation techniques to effectively align the model to the values present in the unstructured data. Through two distinct use-cases, we demonstrate the efficiency of our methodology on the Mistral-7B-Instruct model. Our approach credibly aligns LLMs to the values embedded within documents, and shows improved performance against other approaches, as quantified through the use of automatic metrics and win rates.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 22 of its 26 references that are in the map. First posted on arXiv 19 Aug 2024.

Builds on

22 of 26 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Alignment Studio: Aligning Large Language Models to Particular Contextual Regulations8 Mar
  2. 2024A Roadmap to Pluralistic Alignment7 Feb
  3. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  4. 2020Learning to summarize from human feedback2 Sep
  5. 2020Aligning AI With Shared Human Values5 Aug

Other references · 17

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  7. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  8. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  9. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  10. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  11. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  12. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
5 more
  1. 2024Knowledge Conflicts for LLMs: A Surveycited by 274
  2. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 121
  3. 2024ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidencecited by 109
  4. 2024Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Modelscited by 99
  5. 2023The History and Risks of Reinforcement Learning and Human Feedbackcited by 42

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Agentic Design Patterns: A System-Theoretic Frameworkcited by 4
  2. 2026"I Don't Know" -- Towards Appropriate Trust with Certainty-Aware Retrieval Augmented Generationcited by 0
  3. 2026When Synthetic Speech Is All You Have: Better Call GRPOcited by 0
  4. 2026Granite.Trust Policy Tools: Shareable, Actionable Policies for Generative AI Applicationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026VALUEFLOW: Toward Pluralistic and Steerable Value-based Alignment in Large Language Models
  2. 2026VC-Soup: Value-Consistency Guided Multi-Value Alignment for Large Language Models
  3. 2025Multi-Value Alignment for LLMs via Value Decorrelation and Extrapolation
  4. 2025Internal Value Alignment in Large Language Models through Controlled Value Vector Activation
  5. 2023Heterogeneous Value Alignment Evaluation for Large Language Models
  6. 2023Aligning Large Language Models with Human: A Survey
  7. 2026VISA: Value Injection via Shielded Adaptation for Personalized LLM Alignment
  8. 2025DiverValue-Bench: A Benchmark and Fine-Tuning Framework for Aligning Large Language Models with Diverse Human Values