RTLC -- Research, Teach-to-Learn, Critique: A three-stage prompting paradigm inspired by the Feynman Learning Technique that lifts LLM-as-judge accuracy on JudgeBench with no fine-tuning
Not on the map. Why?
Abstract (arXiv)
LLM-as-a-judge is now the default measurement instrument for open-ended generation, but on the public JudgeBench benchmark even strong instruction-tuned judges barely scrape past random on objective-correctness pairwise items. We introduce RTLC, a three-stage prompting recipe -- Research, Teach-to-Learn, Critique -- that promotes a single black-box LLM into an ensemble-of-thought judge with no fine-tuning, retrieval, or external tools. Stage 1 wraps the input in a fixed pedagogical scaffold porting the Feynman Learning Technique (study → teach → find gaps → simplify) into LLM prompting. Stage 2 draws N=10 independent candidate verdicts at temperature 0.4. Stage 3 acts as its own critic, cross-comparing the candidate set against the original question to emit one critiqued verdict at temperature 0. On JudgeBench-GPT (350 hard pairwise items), Claude 3.7 Sonnet's pairwise accuracy climbs from 64.6% (single-shot vanilla prompt) to 78.6% (RTLC critique-of-10) -- an absolute 14.0-percentage-point gain. RTLC also beats N=10 self-consistency majority voting (77.7%) and a zero-shot first candidate (74.0%). A clean three-step ablation attributes +9.4 pp to the Teach-to-Learn scaffold, +3.7 pp to N=10 marginalisation, and +0.9 pp to explicit critique. We discuss the cost-accuracy frontier (RTLC sits above self-consistency at every working point), the error-budget breakdown across the four JudgeBench categories (knowledge, reasoning, math, coding), and how RTLC composes orthogonally with post-hoc judge-score calibration, with the two interventions compounding multiplicatively in practice.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 10 of its 12 references that are in the map. First posted on arXiv 13 May 2026.
Builds on
10 of 12 references.
References · 10
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
- 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
- 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
- 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
- 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
- 2026Two Ways to De-Bias an LLM-as-a-Judge: A Continuous-Score Comparison of Hierarchical Bayesian Calibration and Neural-ODE Score Transportcited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators
- 2026Training LLM Judges from Language Feedback via Position-Selective Self-Distillation
- 2026Radioactive Feedback: Tracing Models Trained by Label-Only LLM Supervision
- 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning
- 2024CriticBench: Benchmarking LLMs for Critique-Correct Reasoning
- 2025You Don't Need Prompt Engineering Anymore: The Prompting Inversion
- 2025Verdict: A Library for Scaling Judge-Time Compute
- 2026Self-Evaluation Is Already There: Eliciting Latent Judge Calibration in Base LLMs with Minimal Data