mlsanityData to 7 Oct 2026

Second Thoughts are Best: Learning to Re-Align With Human Values from Text Edits

arXiv 2301.00355 ↗First posted 1 Jan 2023cs.CL cs.AI cs.CYcited by 27

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

We present Second Thought, a new learning paradigm that enables language models (LMs) to re-align with human values. By modeling the chain-of-edits between value-unaligned and value-aligned text, with LM fine-tuning and additional refinement through reinforcement learning, Second Thought not only achieves superior performance in three value alignment benchmark datasets but also shows strong human-value transfer learning ability in few-shot scenarios. The generated editing steps also offer better interpretability and ease for interactive error correction. Extensive human evaluations further confirm its effectiveness.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 60 of its 92 references that are in the map. First posted on arXiv 1 Jan 2023.

Builds on

60 of 92 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  2. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 58

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2017Deep reinforcement learning from human preferencescited by 3,846
  11. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  12. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
38 more
  1. 2020Learning to summarize from human feedbackcited by 2,562
  2. 2015Trust Region Policy Optimizationcited by 2,503
  3. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  5. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  6. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  7. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  8. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  9. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  10. 2022Red Teaming Language Models with Language Modelscited by 983
  11. 2021Ethical and social risks of harm from Language Modelscited by 957
  12. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  13. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  14. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  15. 2020Aligning AI With Shared Human Valuescited by 706
  16. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  17. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  18. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  19. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 377
  20. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 343
  21. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 336
  22. 2020Autoregressive Entity Retrievalcited by 333
  23. 2021Unsolved Problems in ML Safetycited by 291
  24. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  25. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  26. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  27. 2020Social Chemistry 101: Learning to Reason about Social and Moral Normscited by 253
  28. 2022Can language models learn from explanations in context?cited by 248
  29. 2022CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilitiescited by 245
  30. 2019Levenshtein Transformercited by 197
  31. 2020WT5?! Training Text-to-Text Models to Explain their Predictionscited by 148
  32. 2020Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequencescited by 133
  33. 2022CommonsenseQA 2.0: Exposing the Limits of AI through Gamificationcited by 108
  34. 2020Scruples: A Corpus of Community Ethical Judgments on 32,000 Real-Life Anecdotescited by 100
  35. 2019Encode, Tag, Realize: High-Precision Text Editingcited by 99
  36. 2022The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systemscited by 92
  37. 2018Hierarchical Imitation and Reinforcement Learningcited by 88
  38. 2021Few-Shot Self-Rationalization with Natural Language Promptscited by 83

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 11; 2024: 10; 2025: 4; 2026: 2.

27 papers.

Sort:

By area:

  1. 2023Large Language Model Alignment: A Surveycited by 241
  2. 2024TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Spacecited by 90
  3. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  4. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedbackcited by 86
  5. 2024Aligning LLM Agents by Learning Latent Preference from User Editscited by 66
  6. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
  7. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Valuescited by 53
  8. 2024Can AI writing be salvaged? Mitigating Idiosyncrasies and Improving Human-AI Alignment in the Writing Process through Editscited by 49
  9. 2023Learning from models beyond fine-tuningcited by 47
  10. 2024TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedbackcited by 43
  11. 2023Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignmentcited by 35
  12. 2023The Touch'e23-ValueEval Dataset for Identifying Human Values behind Argumentscited by 27
15 more
  1. 2024A Survey on Human Preference Learning for Large Language Modelscited by 23
  2. 2024On the Essence and Prospect: An Investigation of Alignment Approaches for Big Modelscited by 21
  3. 2023Enhancing Chain-of-Thoughts Prompting with Iterative Bootstrapping in Large Language Modelscited by 16
  4. 2023Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvementscited by 14
  5. 2024CMDAG: A Chinese Metaphor Dataset with Annotated Grounds as CoT for Boosting Metaphor Generationcited by 11
  6. 2024COPR: Continual Human Preference Learning via Optimal Policy Regularizationcited by 8
  7. 2023Subjective Crowd Disagreements for Subjective Data: Uncovering Meaningful CrowdOpinion with Population-level Learningcited by 7
  8. 2023The Empty Signifier Problem: Towards Clearer Paradigms for Operationalising "Alignment" in Large Language Modelscited by 7
  9. 2025Evaluating and Aligning Human Economic Risk Preferences in LLMscited by 7
  10. 2025SPRI: Aligning Large Language Models with Context-Situated Principlescited by 5
  11. 2023Personalized Abstractive Summarization by Tri-agent Generation Pipelinecited by 4
  12. 2026D2VBench: Benchmarking Large Language Models with Value Dilemmas in Daily Scenarioscited by 1
  13. 2025A Closer Look at Adversarial Suffix Learning for Jailbreaking LLMs: Augmented Adversarial Trigger Learningcited by 0
  14. 2025CALMA: A Process for Deriving Context-aligned Axes for Language Model Alignmentcited by 0
  15. 2026SPHERE: Adaptive VR Indoor Scene Generation via LLM-Enhanced Spatial Preference Learning and Human-in-the-Loop RLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026VISA: Value Injection via Shielded Adaptation for Personalized LLM Alignment
  2. 2025Multi-Value Alignment for LLMs via Value Decorrelation and Extrapolation
  3. 2026Toward Stable Value Alignment: Introducing Independent Modules for Consistent Value Guidance
  4. 2026VC-Soup: Value-Consistency Guided Multi-Value Alignment for Large Language Models
  5. 2026Controllable Value Alignment in Large Language Models through Neuron-Level Editing
  6. 2026VALUEFLOW: Toward Pluralistic and Steerable Value-based Alignment in Large Language Models
  7. 2024Are the Values of LLMs Structurally Aligned with Humans? A Causal Perspective
  8. 2024Embedding an Ethical Mind: Aligning Text-to-Image Synthesis via Lightweight Value Optimization