mlsanityData to 7 Oct 2026

The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning

arXiv 2312.01552 ↗First posted 4 Dec 2023cs.CL cs.AIcited by 251

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

The alignment tuning process of large language models (LLMs) typically involves instruction learning through supervised fine-tuning (SFT) and preference tuning via reinforcement learning from human feedback (RLHF). A recent study, LIMA (Zhou et al. 2023), shows that using merely 1K examples for SFT can achieve significant alignment performance as well, suggesting that the effect of alignment tuning might be "superficial." This raises questions about how exactly the alignment tuning transforms a base LLM. We analyze the effect of alignment tuning by examining the token distribution shift between base LLMs and their aligned counterpart. Our findings reveal that base LLMs and their alignment-tuned versions perform nearly identically in decoding on the majority of token positions. Most distribution shifts occur with stylistic tokens. These direct evidence strongly supports the Superficial Alignment Hypothesis suggested by LIMA. Based on these findings, we rethink the alignment of LLMs by posing the research question: how effectively can we align base LLMs without SFT or RLHF? To address this, we introduce a simple, tuning-free alignment method, URIAL. URIAL achieves effective alignment purely through in-context learning (ICL) with base LLMs, requiring as few as three constant stylistic examples and a system prompt. We conduct a fine-grained and interpretable evaluation on a diverse set of examples, named JUST-EVAL-INSTRUCT. Results demonstrate that base LLMs with URIAL can match or even surpass the performance of LLMs aligned with SFT or SFT+RLHF. We show that the gap between tuning-free and tuning-based alignment methods can be significantly reduced through strategic prompting and ICL. Our findings on the superficial nature of alignment tuning and results with URIAL suggest that deeper analysis and theoretical understanding of alignment is crucial to future LLM research.

Key facts. Cited by 251 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 43 of its 51 references that are in the map. First posted on arXiv 4 Dec 2023.

Builds on

43 of 51 references.

References · 43

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  11. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  12. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
31 more
  1. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  2. 2017Billion-scale similarity search with GPUscited by 2,477
  3. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  5. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  6. 2023LIMA: Less Is More for Alignmentcited by 1,080
  7. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  8. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  9. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  10. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  11. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  12. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  13. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  14. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  15. 2023Larger language models do in-context learning differentlycited by 386
  16. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  17. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  18. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  19. 2023The False Promise of Imitating Proprietary LLMscited by 254
  20. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  21. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  22. 2022Active Example Selection for In-Context Learningcited by 195
  23. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 190
  24. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  25. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 149
  26. 2023Agent Lumos: Unified and Modular Training for Open-Source Language Agentscited by 72
  27. 2023TIGERScore: Towards Building Explainable Metric for All Text Generation Taskscited by 69
  28. 2022Unsupervised Cross-Task Generalization via Retrieval Augmentationcited by 42
  29. 2023The Trickle-down Impact of Reward (In-)consistency on RLHFcited by 29
  30. 2023In-Context Alignment: Chat with Vanilla Language Models Before Fine-Tuningcited by 19
  31. 2023Exploring the Relationship between In-Context Learning and Instruction Tuningcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 113; 2025: 96; 2026: 41.

251 papers.

Sort:

By area:

  1. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  2. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  3. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  4. 2025Search-o1: Agentic Search-Enhanced Large Reasoning Modelscited by 423
  5. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  6. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 233
  7. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 227
  8. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 211
  9. 2024Many-Shot In-Context Learningcited by 203
  10. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 162
  11. 2024Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviewscited by 160
  12. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMscited by 160
38 more
  1. 2024What is in Your Safe Data? Identifying Benign Data that Breaks Safetycited by 114
  2. 2025Small Models Struggle to Learn from Strong Reasonerscited by 109
  3. 2025Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoningcited by 101
  4. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 100
  5. 2024LLaMA Pro: Progressive LLaMA with Block Expansioncited by 99
  6. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 94
  7. 2024How Alignment and Jailbreak Work: Explain LLM Safety through Intermediate Hidden Statescited by 92
  8. 2025Measuring Sycophancy of Language Models in Multi-turn Dialoguescited by 91
  9. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89
  10. 2024Tuning Language Models by Proxycited by 88
  11. 2024Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solvingcited by 82
  12. 2024InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidancecited by 78
  13. 2024Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalizationcited by 78
  14. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editingcited by 77
  15. 2024A Closer Look at the Limitations of Instruction Tuningcited by 76
  16. 2024On the Role of Attention Heads in Large Language Model Safetycited by 70
  17. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 68
  18. 2024Aligning Large Language Models with Representation Editing: A Control Perspectivecited by 62
  19. 2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Samplingcited by 61
  20. 2024Do LLMs Have Distinct and Consistent Personality? TRAIT: Personality Testset designed for LLMs with Psychometricscited by 56
  21. 2024PersonalLLM: Tailoring LLMs to Individual Preferencescited by 56
  22. 2024Learning diverse attacks on large language models for robust red-teaming and safety tuningcited by 54
  23. 2024Jailbreak Large Vision-Language Models Through Multi-Modal Linkagecited by 53
  24. 2025Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoningcited by 50
  25. 2024The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarizationcited by 49
  26. 2025Base Models Beat Aligned Models at Randomness and Creativitycited by 48
  27. 2024Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulationcited by 46
  28. 2025A Survey on the Optimization of Large Language Model-based Agentscited by 45
  29. 2024A Theoretical Understanding of Self-Correction through In-context Alignmentcited by 43
  30. 2025Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedbackcited by 41
  31. 2024Navigating the OverKill in Large Language Modelscited by 40
  32. 2024Cascade Reward Sampling for Efficient Decoding-Time Alignmentcited by 38
  33. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 37
  34. 2024Emulated Disalignment: Safety Alignment for Large Language Models May Backfire!cited by 36
  35. 2024Personality Alignment of Large Language Modelscited by 36
  36. 2024Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirementscited by 35
  37. 2025The Homogenizing Effect of Large Language Models on Human Expression and Thoughtcited by 35
  38. 2024Model Editing as a Robust and Denoised variant of DPO: A Case Study on Toxicitycited by 33

50 most cited of 251 All 251 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Aligner: One Global Token is Worth Millions of Parameters When Aligning Large Language Models
  2. 2025Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Models
  3. 2024Supervised Fine-Tuning as Inverse Reinforcement Learning
  4. 2024Your Weak LLM is Secretly a Strong Teacher for Alignment
  5. 2025Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment
  6. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  7. 2025A Comprehensive Evaluation framework of Alignment Techniques for LLMs
  8. 2023Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment