# Distilled Self-Critique of LLMs with Synthetic Data: a Bayesian Perspective

arXiv: https://arxiv.org/abs/2312.01957 · HTML: https://mlsanity.com/paper/2312.01957/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 10 of its 15 references that are in the map.
- First posted on arXiv 4 Dec 2023.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)
- Synthetic data and self-improvement for LLMs / [LLM synthetic data generation](https://mlsanity.com/synth/llm-synthetic-data-generation/)

## Builds on (most cited first, 10 of 10)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Textbooks Are All You Need II: phi-1.5 technical report](https://mlsanity.com/paper/2309.05463/) (2023)
- [Reinforced Self-Training (ReST) for Language Modeling](https://mlsanity.com/paper/2308.08998/) (2023)
- [RL with KL penalties is better viewed as Bayesian inference](https://mlsanity.com/paper/2205.11275/) (2022)
- [ZYN: Zero-Shot Reward Models with Yes-No Questions for RLAIF](https://mlsanity.com/paper/2308.06385/) (2023)

## Built on by (most cited first, 5 of 5)

- [Efficient Reinforcement Learning with Large Language Model Priors](https://mlsanity.com/paper/2410.07927/) (2024)
- [Configurable Safety Tuning of Language Models with Synthetic Preference Data](https://mlsanity.com/paper/2404.00495/) (2024)
- [Refined Direct Preference Optimization with Synthetic Data for Behavioral Alignment of LLMs](https://mlsanity.com/paper/2402.08005/) (2024)
- [Merging Improves Self-Critique Against Jailbreak Attacks](https://mlsanity.com/paper/2406.07188/) (2024)
- [Moltbook Moderation: Uncovering Hidden Intent Through Multi-Turn Dialogue](https://mlsanity.com/paper/2605.12856/) (2026)

## Abstract (arXiv)

This paper proposes an interpretation of RLAIF as Bayesian inference by introducing distilled Self-Critique (dSC), which refines the outputs of a LLM through a Gibbs sampler that is later distilled into a fine-tuned model. Only requiring synthetic data, dSC is exercised in experiments regarding safety, sentiment, and privacy control, showing it can be a viable and cheap alternative to align LLMs. Code released at urlhttps://github.com/vicgalle/distilled-self-critique.