mlsanityData to 14 Oct 2026

A Theoretical Understanding of Self-Correction through In-context Alignment

arXiv 2405.18634 ↗First posted 28 May 2024cs.LG cs.CL stat.MLcited by 44

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Going beyond mimicking limited human experiences, recent studies show initial evidence that, like humans, large language models (LLMs) are capable of improving their abilities purely by self-correction, i.e., correcting previous responses through self-examination, in certain circumstances. Nevertheless, little is known about how such capabilities arise. In this work, based on a simplified setup akin to an alignment task, we theoretically analyze self-correction from an in-context learning perspective, showing that when LLMs give relatively accurate self-examinations as rewards, they are capable of refining responses in an in-context way. Notably, going beyond previous theories on over-simplified linear transformers, our theoretical construction underpins the roles of several key designs of realistic transformers for self-correction: softmax attention, multi-head attention, and the MLP block. We validate these findings extensively on synthetic datasets. Inspired by these findings, we also illustrate novel applications of self-correction, such as defending against LLM jailbreaks, where a simple self-correction step does make a large difference. We believe that these findings will inspire further research on understanding, exploiting, and enhancing self-correction for building better foundation models.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 79 of its 97 references that are in the map. First posted on arXiv 28 May 2024.

Builds on

79 of 97 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Small Language Models Need Strong Verifiers to Self-Correct Reasoning26 Apr
  2. 2024CriticBench: Benchmarking LLMs for Critique-Correct Reasoning22 Feb
  3. 2023LLMs cannot find reasoning errors, but can correct them given the error location14 Nov
  4. 2023Large Language Models Cannot Self-Correct Reasoning Yet3 Oct
  5. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies6 Aug
  6. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Models17 May
  7. 2023REFINER: Reasoning Feedback on Intermediate Representations4 Apr
  8. 2023Self-Refine: Iterative Refinement with Self-Feedback30 Mar

Other references · 71

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  10. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  12. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
38 more
  1. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  2. 2023Teaching Large Language Models to Self-Debugcited by 988
  3. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  4. 2022Transformers learn in-context by gradient descentcited by 732
  5. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  6. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  7. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
  8. 2019Symmetric Cross Entropy for Robust Learning with Noisy Labelscited by 500
  9. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 498
  10. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
  11. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
  12. 2023Detecting Language Model Attacks with Perplexitycited by 431
  13. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 431
  14. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 423
  15. 2023Language Models can Solve Computer Taskscited by 372
  16. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  17. 2023Low-Resource Languages Jailbreak GPT-4cited by 309
  18. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  19. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 287
  20. 2023Preference Ranking Optimization for Human Alignmentcited by 285
  21. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  22. 2023Certifying LLM Safety against Adversarial Promptingcited by 266
  23. 2020Normalized Loss Functions for Deep Learning with Noisy Labelscited by 257
  24. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 257
  25. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 240
  26. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 222
  27. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 199
  28. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLMcited by 192
  29. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 173
  30. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 157
  31. 2022Transformer Language Models without Positional Encodings Still Learn Positional Informationcited by 150
  32. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Modelscited by 134
  33. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 130
  34. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 118
  35. 2023In-Context Convergence of Transformerscited by 115
  36. 2023Are Emergent Abilities in Large Language Models just In-Context Learning?cited by 109
  37. 2023Can Large Language Models Really Improve by Self-critiquing Their Own Plans?cited by 108
  38. 2022Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlapcited by 105

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 27; 2026: 10.

44 papers.

Sort:

By area:

  1. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 240
  2. 2025S*: Test Time Scaling for Code Generationcited by 98
  3. 2025Efficient Test-Time Scaling via Self-Calibrationcited by 75
  4. 2025PonderLM: Pretraining Language Models to Ponder in Continuous Spacecited by 44
  5. 2025SaLoRA: Safety-Alignment Preserved Low-Rank Adaptationcited by 42
  6. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 40
  7. 2025Reinforced Visual Perception with Toolscited by 38
  8. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 33
  9. 2024Can In-context Learning Really Generalize to Out-of-distribution Tasks?cited by 23
  10. 2025Can LLMs Correct Themselves? A Benchmark of Self-Correction in LLMscited by 14
  11. 2025SAC-Opt: Semantic Anchors for Iterative Correction in Optimization Modelingcited by 8
  12. 2025MetaDefense: Defending Finetuning-based Jailbreak Attack Before and During Generationcited by 8
32 more
  1. 2025Towards the Worst-case Robustness of Large Language Modelscited by 7
  2. 2025Scalable Defense against In-the-wild Jailbreaking Attacks with Safety Context Retrievalcited by 7
  3. 2025HEAL: An Empirical Study on Hallucinations in Embodied Agents Driven by Large Language Modelscited by 6
  4. 2025When and How Unlabeled Data Provably Improve In-Context Learningcited by 6
  5. 2025Theoretical Modeling of Large Language Model Self-Improvement Training Dynamics Through Solver-Verifier Gapcited by 6
  6. 2025Specification Self-Correction: Mitigating In-Context Reward Hacking Through Test-Time Refinementcited by 6
  7. 2024Task Diversity Shortens the ICL Plateaucited by 5
  8. 2025WikiAutoGen: Towards Multi-Modal Wikipedia-Style Article Generationcited by 5
  9. 2025LANPO: Bootstrapping Language and Numerical Feedback for Reinforcement Learning in LLMscited by 5
  10. 2025Self-Reflective Planning with Knowledge Graphs: Enhancing LLM Reasoning Reliability for Question Answeringcited by 4
  11. 2025Long-Short Alignment for Effective Long-Context Modeling in LLMscited by 4
  12. 2025Discourse Heuristics For Paradoxically Moral Self-Correctioncited by 3
  13. 2026Self-Debias: Self-correcting for Debiasing Large Language Modelscited by 3
  14. 2026PiCA: Pivot-Based Credit Assignment for Search Agentic Reinforcement Learningcited by 3
  15. 2024MILE: A Mutation Testing Framework of In-Context Learning Systemscited by 2
  16. 2024Self-correction is Not An Innate Capability in Language Modelscited by 2
  17. 2025Explaining Intrinsic Moral Self-Correction with Mechanistic Interpretabilitycited by 2
  18. 2025Scalable and Robust LLM Unlearning by Correcting Responses with Retrieved Exclusionscited by 2
  19. 2026Finding and Reactivating Post-Trained LLMs' Hidden Safety Mechanismscited by 2
  20. 2026Layered LLM Defenses as an Ensemble: Access Tiers, Inference Cost, and the Measured Failure Correlation Between Defense Layerscited by 2
  21. 2024Past Meets Present: Creating Historical Analogy with Large Language Modelscited by 1
  22. 2025Research Superalignment Should Advance Now with Alternating Competence and Conformity Optimizationcited by 1
  23. 2025ReGA: Model-Based Safeguard for LLMs via Representation-Guided Abstractioncited by 1
  24. 2025CIMR: Contextualized Iterative Multimodal Reasoning for Robust Instruction Following in LVLMscited by 1
  25. 2025An Explainable Emotion Alignment Framework for LLM-Empowered Agent in Metaverse Service Ecosystemcited by 1
  26. 2026Beyond In-Domain Detection: SpikeScore for Cross-Domain Hallucination Detectioncited by 1
  27. 2026Fairness Aware Reward Optimizationcited by 1
  28. 2026Intra-Fairness Dynamics: The Bias Spillover Effect in Targeted LLM Alignmentcited by 1
  29. 2025DynaQuery: A Self-Adapting Framework for Querying Structured and Multimodal Datacited by 0
  30. 2026Though Language Models Err While They Strive: Conformal Prediction for Self-Correcting Scientific Generationcited by 0
  31. 2026Optimal Stopping of Self-Refining Foundation Modelscited by 0
  32. 2026From Static Policies to Adaptive Priors in Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Large Language Models have Intrinsic Self-Correction Ability
  2. 2025Self-Correction Bench: Uncovering and Addressing the Self-Correction Blind Spot in Large Language Models
  3. 2024Confidence v.s. Critique: A Decomposition of Self-Correction Capability for LLMs
  4. 2026When Does Intrinsic Self-Correction Help? A Task-Sensitive Analysis
  5. 2024Smaller Large Language Models Can Do Moral Self-Correction
  6. 2024Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models
  7. 2024Self-Correction is More than Refinement: A Learning Framework for Visual and Language Reasoning Tasks
  8. 2025Intent-Aware Self-Correction for Mitigating Social Biases in Large Language Models