Distilled Self-Critique of LLMs with Synthetic Data: a Bayesian Perspective
On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment
Abstract (arXiv)
This paper proposes an interpretation of RLAIF as Bayesian inference by introducing distilled Self-Critique (dSC), which refines the outputs of a LLM through a Gibbs sampler that is later distilled into a fine-tuned model. Only requiring synthetic data, dSC is exercised in experiments regarding safety, sentiment, and privacy control, showing it can be a viable and cheap alternative to align LLMs. Code released at urlhttps://github.com/vicgalle/distilled-self-critique.
Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 10 of its 15 references that are in the map. First posted on arXiv 4 Dec 2023.
Builds on
10 of 15 references.
References · 10
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
- 2016Generative Adversarial Imitation Learningcited by 1,597
- 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
- 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
- 2022RL with KL penalties is better viewed as Bayesian inferencecited by 135
- 2023ZYN: Zero-Shot Reward Models with Yes-No Questions for RLAIFcited by 4
Built on by
Citing papers per year: 2024: 4; 2025: 0; 2026: 1.
5 papers.
- 2024Efficient Reinforcement Learning with Large Language Model Priorscited by 30
- 2024Configurable Safety Tuning of Language Models with Synthetic Preference Datacited by 14
- 2024Refined Direct Preference Optimization with Synthetic Data for Behavioral Alignment of LLMscited by 10
- 2024Merging Improves Self-Critique Against Jailbreak Attackscited by 8
- 2026Moltbook Moderation: Uncovering Hidden Intent Through Multi-Turn Dialoguecited by 0
- 2026Moltbook Moderation: Uncovering Hidden Intent Through Multi-Turn Dialogue13 May 2026
- 2024Efficient Reinforcement Learning with Large Language Model Priors10 Oct 2024
- 2024Merging Improves Self-Critique Against Jailbreak Attacks11 Jun 2024
- 2024Configurable Safety Tuning of Language Models with Synthetic Preference Data30 Mar 2024
- 2024Refined Direct Preference Optimization with Synthetic Data for Behavioral Alignment of LLMs12 Feb 2024
Similar, not cited
Close in wording, no citation either way
- 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold
- 2024A Synthetic Dataset for Personal Attribute Inference
- 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls
- 2025LLMs are Overconfident: Evaluating Confidence Interval Calibration with FermiEval
- 2025Self-Consistency Is Losing Its Edge: Diminishing Returns and Rising Costs in Modern LLMs
- 2025RoSE: Round-robin Synthetic Data Evaluation for Selecting LLM Generators without Human Test Sets
- 2025A Technical Exploration of Causal Inference with Hybrid LLM Synthetic Data
- 2026When Are LLM Inferences Acceptable? User Reactions and Control Preferences for Inferred Personal Information