mlsanityData to 7 Oct 2026

Aligning Language Models with Demonstrated Feedback

arXiv 2406.00888 ↗First posted 2 Jun 2024cs.CL cs.HCcited by 32

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Language models are aligned to emulate the collective voice of many, resulting in outputs that align with no one in particular. Steering LLMs away from generic output is possible through supervised finetuning or RLHF, but requires prohibitively large datasets for new ad-hoc tasks. We argue that it is instead possible to align an LLM to a specific setting by leveraging a very small number (< 10) of demonstrations as feedback. Our method, Demonstration ITerated Task Optimization (DITTO), directly aligns language model outputs to a user's demonstrated behaviors. Derived using ideas from online imitation learning, DITTO cheaply generates online comparison data by treating users' demonstrations as preferred over output from the LLM and its intermediate checkpoints. Concretely, DITTO operates by having an LLM generate examples that are presumed to be inferior to expert demonstrations. The method iteratively constructs pairwise preference relationships between these LLM-generated samples and expert demonstrations, potentially including comparisons between different training checkpoints. These constructed preference pairs are then used to train the model using a preference optimization algorithm (e.g. DPO). We evaluate DITTO's ability to learn fine-grained style and task alignment across domains such as news articles, emails, and blog posts. Additionally, we conduct a user study soliciting a range of demonstrations from participants (N = 16). Across our benchmarks and user study, we find that win-rates for DITTO outperform few-shot prompting, supervised fine-tuning, and other self-play methods by an avg. of 19% points. By using demonstrations as feedback directly, DITTO offers a novel method for effective customization of LLMs.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 48 of its 67 references that are in the map. First posted on arXiv 2 Jun 2024.

Builds on

48 of 67 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  2. 2024From r to Q^*: Your Language Model is Secretly a Q-Function18 Apr
  3. 2024Unintended Impacts of LLM Alignment on Global Representation22 Feb
  4. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  5. 2023Group Preference Optimization: Few-Shot Alignment of Large Language Models17 Oct
  6. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 42

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2023Mistral 7Bcited by 3,065
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
30 more
  1. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  2. 2016Generative Adversarial Imitation Learningcited by 1,576
  3. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  4. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  5. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  6. 2023Whose Opinions Do Language Models Reflect?cited by 759
  7. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  8. 2024Self-Rewarding Language Modelscited by 654
  9. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  10. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  11. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  12. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  13. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  14. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  15. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  16. 2023Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Modelscited by 204
  17. 2024Trial and Error: Exploration-Based Trajectory Optimization for LLM Agentscited by 194
  18. 2020Reformulating Unsupervised Style Transfer as Paraphrase Generationcited by 181
  19. 2023Art or Artifice? Large Language Models and the False Promise of Creativitycited by 168
  20. 2024STaR-GATE: Teaching Language Models to Ask Clarifying Questionscited by 99
  21. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedbackcited by 86
  22. 2024Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooMcited by 77
  23. 2024Aligning LLM Agents by Learning Latent Preference from User Editscited by 66
  24. 2018Learning to Extract Coherent Summary via Deep Reinforcement Learningcited by 63
  25. 2023Coherent Soft Imitation Learningcited by 21
  26. 2022A Ranking Game for Imitation Learningcited by 18
  27. 2024RLVF: Learning from Verbal Feedback without Overgeneralizationcited by 18
  28. 2023STEER: Unified Style Transfer with Expert Reinforcementcited by 14
  29. 2024Authorship Style Transfer with Policy Optimizationcited by 11
  30. 2024Model Editing with Canonical Examplescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 24; 2026: 4.

32 papers.

Sort:

By area:

  1. 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMscited by 142
  2. 2025Creating General User Models from Computer Usecited by 44
  3. 2025LoRe: Personalizing LLMs via Low-Rank Reward Modelingcited by 34
  4. 2025SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMscited by 26
  5. 2025What's In My Human Feedback? Learning Interpretable Descriptions of Preference Datacited by 22
  6. 2024AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrationscited by 19
  7. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  8. 2025Beyond the Prompt: An Empirical Study of Cursor Rulescited by 7
  9. 2024PersLLM: A Personified Training Approach for Large Language Modelscited by 6
  10. 2024ChainBuddy: An AI Agent System for Generating LLM Pipelinescited by 6
  11. 2025POET: Supporting Prompting Creativity and Personalization with Automated Expansion of Text-to-Image Generationcited by 6
  12. 2025DRIFT: Learning from Abundant User Dissatisfaction in Real-World Preference Learningcited by 5
20 more
  1. 2025Tuning-Free Personalized Alignment via Trial-Error-Explain In-Context Learningcited by 4
  2. 2025Ontologies in Design: How Imagining a Tree Reveals Possibilites and Assumptions in Large Language Modelscited by 3
  3. 2025Semantic Commit: Helping Users Update Intent Specifications for AI Memory at Scalecited by 3
  4. 2025Moment- and Power-Spectrum-Based Gaussianity Regularization for Text-to-Image Modelscited by 3
  5. 2025Improving User Interface Generation Models from Designer Feedbackcited by 3
  6. 2025Not My Agent, Not My Boundary? Elicitation of Personal Privacy Boundaries in AI-Delegated Information Sharingcited by 3
  7. 2025ALLOY: Generating Reusable Agent Workflows from User Demonstrationcited by 3
  8. 2024Open-domain Implicit Format Control for Large Language Model Generationcited by 2
  9. 2025Policy-labeled Preference Learning: Is Preference Enough for RLHF?cited by 2
  10. 2025Interactive Multi-Objective Probabilistic Preference Learning with Soft and Hard Boundscited by 2
  11. 2025Scaffolding Recursive Divergence and Convergence in Story Ideationcited by 2
  12. 2025Culture Cartography: Mapping the Landscape of Cultural Knowledgecited by 2
  13. 2025Steerable Chatbots: Exploring Personalization Control Interfaces via LLM Activation Steeringcited by 1
  14. 2025Beyond Imitation: Recovering Dense Rewards from Demonstrationscited by 1
  15. 2025Alchemist: Towards the Design of Efficient Online Continual Learning Systemcited by 0
  16. 2025Matching Game Preferences Through Dialogical Large Language Models: A Perspectivecited by 0
  17. 2026In-Context Reward Adaptation for Robust Preference Modelingcited by 0
  18. 2026On the Limits of Steering Vectors for Preference-Aligned Generationcited by 0
  19. 2026Efficient Test-Time Adaptation through Human-AI Interactioncited by 0
  20. 2026Ownership in AI-Assisted Everyday Taskscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Training Language Models with Language Feedback at Scale
  2. 2024Supervised Fine-Tuning as Inverse Reinforcement Learning
  3. 2024Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learning
  4. 2021A General Language Assistant as a Laboratory for Alignment
  5. 2023Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models
  6. 2024Aligning Large Language Models via Fine-grained Supervision
  7. 2024A Systematic Examination of Preference Learning through the Lens of Instruction-Following
  8. 2024LIONs: An Empirically Optimized Approach to Align Language Models