mlsanityData to 13 Oct 2026

RAIN: Your Language Models Can Align Themselves without Finetuning

arXiv 2309.07124 ↗First posted 13 Sep 2023cs.CLcited by 157

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Large language models (LLMs) often demonstrate inconsistencies with human preferences. Previous research typically gathered human preference data and then aligned the pre-trained models using reinforcement learning or instruction tuning, a.k.a. the finetuning step. In contrast, aligning frozen LLMs without requiring alignment data is more appealing. This work explores the potential of the latter setting. We discover that by integrating self-evaluation and rewind mechanisms, unaligned LLMs can directly produce responses consistent with human preferences via self-boosting. We introduce a novel inference method, Rewindable Auto-regressive INference (RAIN), that allows pre-trained LLMs to evaluate their own generation and use the evaluation results to guide rewind and generation for AI safety. Notably, RAIN operates without the need of extra data for model alignment and abstains from any training, gradient computation, or parameter updates. Experimental results evaluated by GPT-4 and humans demonstrate the effectiveness of RAIN: on the HH dataset, RAIN improves the harmlessness rate of LLaMA 30B from 82% of vanilla inference to 97%, while maintaining the helpfulness rate. On the TruthfulQA dataset, RAIN improves the truthfulness of the already-well-aligned LLaMA-2-chat 13B model by 5%.

Key facts. Cited by 157 papers in the map (data through 13 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 36 of its 47 references that are in the map. First posted on arXiv 13 Sep 2023.

Builds on

36 of 47 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023LIMA: Less Is More for Alignment18 May
  2. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision4 May
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  4. 2022Training language models to follow instructions with human feedback4 Mar
  5. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 31

  1. 2017Proximal Policy Optimization Algorithmscited by 14,592
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,107
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,986
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,317
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,691
  9. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,181
  10. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,064
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,011
  12. 2022Training Compute-Optimal Large Language Modelscited by 2,972
19 more
  1. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,949
  2. 2020Learning to summarize from human feedbackcited by 2,575
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,379
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,139
  5. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,736
  6. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,280
  7. 2022LaMDA: Language Models for Dialog Applicationscited by 1,219
  8. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,201
  9. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 751
  10. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 680
  11. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 647
  12. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 617
  13. 2023Textbooks Are All You Needcited by 566
  14. 2022Improving alignment of dialogue agents via targeted human judgementscited by 529
  15. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 459
  16. 2023Detecting Language Model Attacks with Perplexitycited by 429
  17. 2023Certifying LLM Safety against Adversarial Promptingcited by 263
  18. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 177
  19. 2018Can Neural Machine Translation be Improved with User Feedback?cited by 71

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 12; 2024: 70; 2025: 54; 2026: 21.

157 papers.

Sort:

By area:

  1. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 565
  2. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 483
  3. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 429
  4. 2023Low-Resource Languages Jailbreak GPT-4cited by 306
  5. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 293
  6. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 272
  7. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 255
  8. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 245
  9. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 229
  10. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 227
  11. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 214
  12. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritizationcited by 169
38 more
  1. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 156
  2. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 130
  3. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmeticcited by 112
  4. 2024A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Modelscited by 106
  5. 2024Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Spacecited by 90
  6. 2024InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidancecited by 82
  7. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editingcited by 79
  8. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 79
  9. 2024Safeguarding Large Language Models: A Surveycited by 76
  10. 2024Intention Analysis Makes LLMs A Good Jailbreak Defendercited by 73
  11. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 70
  12. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 70
  13. 2025Kwai Keye-VL 1.5 Technical Reportcited by 70
  14. 2024Speak Out of Turn: Safety Vulnerability of Large Language Models in Multi-turn Dialoguecited by 66
  15. 2024SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Mannercited by 63
  16. 2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Samplingcited by 61
  17. 2024DeAL: Decoding-time Alignment for Large Language Modelscited by 55
  18. 2025Foot-In-The-Door: A Multi-turn Jailbreak for LLMscited by 54
  19. 2026Taming Various Privilege Escalation in LLM-Based Agent Systems: A Mandatory Access Control Frameworkcited by 52
  20. 2024Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMscited by 51
  21. 2023Adversarial Attacks and Defenses in Large Language Models: Old and New Threatscited by 49
  22. 2024Risks of AI Scientists: Prioritizing Safeguarding Over Autonomycited by 49
  23. 2024When LLM Meets DRL: Advancing Jailbreaking Efficiency via DRL-guided Searchcited by 47
  24. 2024JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Frameworkcited by 47
  25. 2025Kwai Keye-VL Technical Reportcited by 47
  26. 2024Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulationcited by 46
  27. 2024A Theoretical Understanding of Self-Correction through In-context Alignmentcited by 44
  28. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignmentcited by 44
  29. 2024Cascade Reward Sampling for Efficient Decoding-Time Alignmentcited by 39
  30. 2023When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Taskscited by 37
  31. 2024Safety Arithmetic: A Framework for Test-time Safety Alignment of Language Models by Steering Parameters and Activationscited by 32
  32. 2023Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learningcited by 30
  33. 2024From Theft to Bomb-Making: The Ripple Effect of Unlearning in Defending Against Jailbreak Attackscited by 29
  34. 2025Amulet: ReAlignment During Test Time for Personalized Preference Adaptation of LLMscited by 29
  35. 2025LLM Agents Should Employ Security Principlescited by 28
  36. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Modelscited by 27
  37. 2024Adversarial Tuning: Defending Against Jailbreak Attacks for LLMscited by 26
  38. 2024Human-Instruction-Free LLM Self-Alignment with Limited Samplescited by 25

50 most cited of 157 All 157 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Aligning Large Language Models via Fully Self-Synthetic Data
  2. 2025Aligning Frozen LLMs by Reinforcement Learning: An Iterative Reweight-then-Optimize Approach
  3. 2024MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time
  4. 2024Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models
  5. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  6. 2024Inference time LLM alignment in single and multidomain preference spectrum
  7. 2024Supervised Fine-Tuning as Inverse Reinforcement Learning
  8. 2023Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Language