mlsanityData to 14 Oct 2026

System-Level Natural Language Feedback

arXiv 2306.13588 ↗First posted 23 Jun 2023cs.CL cs.AIcited by 4

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Natural language (NL) feedback offers rich insights into user experience. While existing studies focus on an instance-level approach, where feedback is used to refine specific examples, we introduce a framework for system-level use of NL feedback. We show how to use feedback to formalize system-level design decisions in a human-in-the-loop-process -- in order to produce better models. In particular this is done through: (i) metric design for tasks; and (ii) language model prompt design for refining model responses. We conduct two case studies of this approach for improving search query and dialog response generation, demonstrating the effectiveness of system-level feedback. We show the combination of system-level and instance-level feedback brings further gains, and that human written instance-level feedback results in more grounded refinements than GPT-3.5 written ones, underlying the importance of human feedback for building systems. We release our code and data at https://github.com/yyy-Apple/Sys-NL-Feedback.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 38 of its 56 references that are in the map. First posted on arXiv 23 Jun 2023.

Builds on

38 of 56 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Training Language Models with Language Feedback at Scale28 Mar
  2. 2022When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels28 Oct
  3. 2022Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback5 Aug
  4. 2022Training Language Models with Language Feedback29 Apr
  5. 2020Learning to summarize from human feedback2 Sep

Other references · 33

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  8. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  11. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  12. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
21 more
  1. 2023Teaching Large Language Models to Self-Debugcited by 988
  2. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  3. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  4. 2023Benchmarking Large Language Models for News Summarizationcited by 445
  5. 2022Self-critiquing models for assisting human evaluatorscited by 380
  6. 2023GPTScore: Evaluate as You Desirecited by 361
  7. 2021Beyond Goldfish Memory: Long-Term Open-Domain Conversationcited by 267
  8. 2021Internet-Augmented Dialogue Generationcited by 217
  9. 2019Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Trainingcited by 110
  10. 2019Learning from Dialogue after Deployment: Feed Yourself, Chatbot!cited by 102
  11. 2022Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completioncited by 102
  12. 2023Improving Code Generation by Training with Natural Language Feedbackcited by 76
  13. 2023ChatGPT as a Factual Inconsistency Evaluator for Text Summarizationcited by 75
  14. 2016Dialogue Learning With Human-In-The-Loopcited by 46
  15. 2023VISAR: A Human-AI Argumentative Writing Assistant with Visual Programming and Rapid Draft Prototypingcited by 46
  16. 2022The CRINGE Loss: Learning what language not to modelcited by 39
  17. 2022DIRECTOR: Generator-Classifiers For Supervised Language Modelingcited by 38
  18. 2021Learning to Repair: Repairing model output errors after deployment using a dynamic memory of feedbackcited by 35
  19. 2016Dialog-based Language Learningcited by 21
  20. 2023Seeing ChatGPT Through Students' Eyes: An Analysis of TikTok Datacited by 7
  21. 2023Academic Writing with GPT-3.5: Reflections on Practices, Efficacy and Transparencycited by 6

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 1.

4 papers.

  1. 2023Large Language Models as Optimizerscited by 864
  2. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 238
  3. 2024Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generationcited by 27
  4. 2023Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Modelscited by 23

Similar, not cited

Close in wording, no citation either way

  1. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation
  2. 2023Let Me Teach You: Pedagogical Foundations of Feedback for Language Models
  3. 2023What if you said that differently?: How Explanation Formats Affect Human Feedback Efficacy and User Perception
  4. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlook
  5. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback
  6. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values
  7. 2025What Makes a Good Natural Language Prompt?
  8. 2026Feedback by Design: Understanding and Overcoming User Feedback Barriers in Conversational Agents