mlsanityData to 14 Oct 2026

Nano: Nested Human-in-the-Loop Reward Learning for Few-shot Language Model Control

arXiv 2211.05750 ↗First posted 10 Nov 2022cs.CLcited by 6

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Pretrained language models have demonstrated extraordinary capabilities in language generation. However, real-world tasks often require controlling the distribution of generated text in order to mitigate bias, promote fairness, and achieve personalization. Existing techniques for controlling the distribution of generated text only work with quantified distributions, which require pre-defined categories, proportions of the distribution, or an existing corpus following the desired distributions. However, many important distributions, such as personal preferences, are unquantified. In this work, we tackle the problem of generating text following arbitrary distributions (quantified and unquantified) by proposing Nano, a few-shot human-in-the-loop training algorithm that continuously learns from human feedback. Nano achieves state-of-the-art results on single topic/attribute as well as quantified distribution control compared to previous works. We also show that Nano is able to learn unquantified distributions, achieves personalization, and captures differences between different individuals' personal preferences with high sample efficiency.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 42 of its 64 references that are in the map. First posted on arXiv 10 Nov 2022.

Builds on

42 of 64 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020Learning to summarize from human feedback2 Sep

Other references · 41

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  4. 2015Continuous control with deep reinforcement learningcited by 3,407
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  7. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 799
  8. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 762
  9. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 541
  10. 2019Neural Text Generation with Unlikelihood Trainingcited by 503
  11. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 455
  12. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 296
29 more
  1. 2016Deep Reinforcement Learning for Dialogue Generationcited by 288
  2. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 237
  3. 2021A Survey of Human-in-the-loop for Machine Learningcited by 215
  4. 2019Identifying and Reducing Gender Bias in Word-Level Language Modelscited by 214
  5. 2017Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluationcited by 206
  6. 2018Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfercited by 191
  7. 2020Towards Debiasing Sentence Representationscited by 171
  8. 2018Style Transfer Through Back-Translationcited by 124
  9. 2020A Distributional Approach to Controlled Text Generationcited by 119
  10. 2020Towards Controllable Biases in Language Generationcited by 112
  11. 2019Learning from Dialogue after Deployment: Feed Yourself, Chatbot!cited by 102
  12. 2017Generating Sentences by Editing Prototypescited by 88
  13. 2019Transforming Delete, Retrieve, Generate Approach for Controlled Text Style Transfercited by 81
  14. 2020Politeness Transfer: A Tag and Generate Approachcited by 78
  15. 2018A Retrieve-and-Edit Framework for Predicting Structured Outputscited by 73
  16. 2020A Probabilistic Formulation of Unsupervised Text Style Transfercited by 70
  17. 2017Paraphrase Generation with Deep Reinforcement Learningcited by 58
  18. 2019Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Modelscited by 53
  19. 2020CoCon: A Self-Supervised Approach for Controlled Text Generationcited by 47
  20. 2019Model-based Interactive Semantic Parsing: A Unified Framework and A Text-to-SQL Case Studycited by 40
  21. 2021Your fairness may vary: Pretrained language model fairness in toxic text classificationcited by 39
  22. 2021StylePTB: A Compositional Benchmark for Fine-grained Controllable Text Style Transfercited by 34
  23. 2019Towards Controllable and Personalized Review Generationcited by 24
  24. 2018Interactive Semantic Parsing for If-Then Recipes via Hierarchical Reinforcement Learningcited by 7
  25. 2019Learning When Not to Answer: A Ternary Reward Structure for Reinforcement Learning based Question Answeringcited by 7
  26. 2021When and Why does a Model Fail? A Human-in-the-loop Error Detection Framework for Sentiment Analysiscited by 4
  27. 2019Personalized sentence generation using generative adversarial networks with author-specific word usagecited by 3
  28. 2019Automatic Conditional Generation of Personalized Social Media Short Textscited by 2
  29. 2021Reinforced Generative Adversarial Network for Abstractive Text Summarizationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 1; 2024: 3; 2025: 0; 2026: 1.

6 papers.

  1. 2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 228
  2. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 90
  3. 2022Reward Gaming in Conditional Text Generationcited by 30
  4. 2024Transforming and Combining Rewards for Aligning Large Language Modelscited by 25
  5. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 18
  6. 2026A Comparative Study of Controlled Text Generation Systems Using Level-Playing-Field Evaluation Principlescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Pretraining Language Models with Human Preferences
  2. 2024A Critical Look At Tokenwise Reward-Guided Text Generation
  3. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training
  4. 2025Language Model Personalization via Reward Factorization
  5. 2024Personalized Language Modeling from Personalized Human Feedback
  6. 2024Aligning language models with human preferences
  7. 2025Zero-Shot LLMs in Human-in-the-Loop RL: Replacing Human Feedback for Reward Shaping
  8. 2023Training Language Models with Language Feedback at Scale