mlsanityData to 7 Oct 2026

Towards Aligning Language Models with Textual Feedback

arXiv 2407.16970 ↗First posted 24 Jul 2024cs.CL cs.AI cs.LGcited by 5

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

We present ALT (ALignment with Textual feedback), an approach that aligns language models with user preferences expressed in text. We argue that text offers greater expressiveness, enabling users to provide richer feedback than simple comparative preferences and this richer feedback can lead to more efficient and effective alignment. ALT aligns the model by conditioning its generation on the textual feedback. Our method relies solely on language modeling techniques and requires minimal hyper-parameter tuning, though it still presents the main benefits of RL-based alignment algorithms and can effectively learn from textual feedback. We explore the efficacy and efficiency of textual feedback across different tasks such as toxicity reduction, summarization, and dialog response generation. We find that ALT outperforms PPO for the task of toxicity reduction while being able to match its performance on summarization with only 20% of the samples. We also explore how ALT can be used with feedback provided by an existing LLM where we explore an LLM providing constrained and unconstrained textual feedback. We also outline future directions to align models with natural language feedback.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 40 references that are in the map. First posted on arXiv 24 Jul 2024.

Builds on

30 of 40 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Training Language Models with Language Feedback at Scale28 Mar
  2. 2022Training Language Models with Language Feedback29 Apr
  3. 2020Learning to summarize from human feedback2 Sep

Other references · 27

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  7. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  8. 2018Hierarchical Neural Story Generationcited by 1,182
  9. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  10. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  11. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  12. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
15 more
  1. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  2. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  3. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
  4. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  5. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
  6. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  7. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
  8. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
  9. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  10. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 124
  11. 2023SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHFcited by 107
  12. 2019Reward-Conditioned Policiescited by 101
  13. 2021Scalar reward is not enough: A response to Silver, Singh, Precup and Sutton (2021)cited by 55
  14. 2023Aligning Language Models with Offline Learning from Human Feedbackcited by 14
  15. 2024How Much Annotation is Needed to Compare Summarization Models?cited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 3.

5 papers.

  1. 2025Recontextualization Mitigates Specification Gaming without Modifying the Specificationcited by 11
  2. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMscited by 10
  3. 2026Structuring the Space of Sociotechnical Alignmentcited by 2
  4. 2026RL with Learnable Textual Feedback: A Bilevel Approachcited by 1
  5. 2026The Rise of Verbal Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Language
  2. 2023Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models
  3. 2025DxHF: Providing High-Quality Human Feedback for LLM Alignment via Interactive Decomposition
  4. 2024MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time
  5. 2024WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback
  6. 2025Fine-Tuning LLMs with Fine-Grained Human Feedback on Text Spans
  7. 2025Aligning Large Language Models with Implicit Preferences from User-Generated Content
  8. 2023Constructive Large Language Models Alignment with Diverse Feedback