mlsanityData to 7 Oct 2026

Shepherd: A Critic for Language Model Generation

arXiv 2308.04592 ↗First posted 8 Aug 2023cs.CL cs.AIcited by 96

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

As large language models improve, there is increasing interest in techniques that leverage these models' capabilities to refine their own outputs. In this work, we introduce Shepherd, a language model specifically tuned to critique responses and suggest refinements, extending beyond the capabilities of an untuned model to identify diverse errors and provide suggestions to remedy them. At the core of our approach is a high quality feedback dataset, which we curate from community feedback and human annotations. Even though Shepherd is small (7B parameters), its critiques are either equivalent or preferred to those from established models including ChatGPT. Using GPT-4 for evaluation, Shepherd reaches an average win-rate of 53-87% compared to competitive alternatives. In human evaluation, Shepherd strictly outperforms other models and on average closely ties with ChatGPT.

Key facts. Cited by 96 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 37 of its 61 references that are in the map. First posted on arXiv 8 Aug 2023.

Builds on

37 of 61 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing19 May
  2. 2023Self-Refine: Iterative Refinement with Self-Feedback30 Mar
  3. 2022Large Language Models Can Self-Improve20 Oct

Other references · 34

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  8. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  10. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  11. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  12. 2017Decoupled Weight Decay Regularizationcited by 1,722
22 more
  1. 2023LIMA: Less Is More for Alignmentcited by 1,080
  2. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  3. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  4. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  5. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  6. 2023Instruction Tuning with GPT-4cited by 676
  7. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  8. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  9. 2022Self-critiquing models for assisting human evaluatorscited by 372
  10. 2023GPTScore: Evaluate as You Desirecited by 357
  11. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  12. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  13. 2023How Language Model Hallucinations Can Snowballcited by 310
  14. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 309
  15. 2020The Pushshift Reddit Datasetcited by 289
  16. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 281
  17. 2022Generating Sequences by Learning to Self-Correctcited by 271
  18. 2022Re3: Generating Longer Stories With Recursive Reprompting and Revisioncited by 193
  19. 2022ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoningcited by 175
  20. 2021Explaining Answers with Entailment Treescited by 148
  21. 2023Can we trust the evaluation on ChatGPT?cited by 63
  22. 2022On Improving Summarization Factual Consistency from Natural Language Feedbackcited by 38

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 17; 2024: 44; 2025: 27; 2026: 8.

96 papers.

Sort:

By area:

  1. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  2. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  3. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  4. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  5. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  6. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 257
  7. 2023Generative Judge for Evaluating Alignmentcited by 163
  8. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizingcited by 123
  9. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  10. 2023Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revisioncited by 96
  11. 2024CriticBench: Benchmarking LLMs for Critique-Correct Reasoningcited by 88
  12. 2023EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteriacited by 73
38 more
  1. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 71
  2. 2023CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generationcited by 65
  3. 2024TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generationcited by 62
  4. 2024Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuningcited by 57
  5. 2024CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferencescited by 56
  6. 2025Reinforcement Learning from Human Feedbackcited by 53
  7. 2023The History and Risks of Reinforcement Learning and Human Feedbackcited by 42
  8. 2024Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Ratescited by 42
  9. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  10. 2025Teaching Language Models to Critique via Reinforcement Learningcited by 38
  11. 2023Thought Propagation: An Analogical Approach to Complex Reasoning with Large Language Modelscited by 37
  12. 2024Leveraging Large Language Models for NLG Evaluation: Advances and Challengescited by 37
  13. 2023Advancements in Generative AI: A Comprehensive Review of GANs, GPT, Autoencoders, Diffusion Model, and Transformerscited by 35
  14. 2024Evaluating the Evaluator: Measuring LLMs' Adherence to Task Evaluation Instructionscited by 34
  15. 2024Direct Judgement Preference Optimizationcited by 33
  16. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 31
  17. 2024Learning to Refine with Fine-Grained Natural Language Feedbackcited by 27
  18. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  19. 2025To Backtrack or Not to Backtrack: When Sequential Search Limits Model Reasoningcited by 24
  20. 2024Task Success is not Enough: Investigating the Use of Video-Language Models as Behavior Critics for Catching Undesirable Agent Behaviorscited by 23
  21. 2024A Survey on Human Preference Learning for Large Language Modelscited by 23
  22. 2024CLAVE: An Adaptive Framework for Evaluating Values of LLM Generated Responsescited by 23
  23. 2024Better than Your Teacher: LLM Agents that learn from Privileged AI Feedbackcited by 21
  24. 2025Sample, Don't Search: Rethinking Test-Time Alignment for Language Modelscited by 21
  25. 2024Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretabilitycited by 19
  26. 2024What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluationcited by 17
  27. 2024Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competitioncited by 16
  28. 2024RichRAG: Crafting Rich Responses for Multi-faceted Queries in Retrieval-Augmented Generationcited by 16
  29. 2023ChatGPT's One-year Anniversary: Are Open-Source Large Language Models Catching up?cited by 15
  30. 2023Reasons to Reject? Aligning Language Models with Judgmentscited by 15
  31. 2024Intrinsic Self-correction for Enhanced Morality: An Analysis of Internal Mechanisms and the Superficial Hypothesiscited by 15
  32. 2024Improving Model Factuality with Fine-grained Critique-based Evaluatorcited by 15
  33. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
  34. 2024MCQG-SRefine: Multiple Choice Question Generation and Evaluation with Iterative Self-Critique, Correction, and Comparison Feedbackcited by 14
  35. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  36. 2024Suppressing Pink Elephants with Direct Principle Feedbackcited by 13
  37. 2025RefineBench: Evaluating Refinement Capability of Language Models via Checklistscited by 13
  38. 2023Adaptive-Solver Framework for Dynamic Strategy Selection in Large Language Model Reasoningcited by 12

50 most cited of 96 All 96 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Self-Evolving Critique Abilities in Large Language Models
  2. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation
  3. 2022Discovering Language Model Behaviors with Model-Written Evaluations
  4. 2022Training Language Models with Language Feedback
  5. 2025Accelerating Unbiased LLM Evaluation via Synthetic Feedback
  6. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlook
  7. 2024Evaluating Large Language Models with Human Feedback: Establishing a Swedish Benchmark
  8. 2023RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs