mlsanityData to 7 Oct 2026

Self-Contrast: Better Reflection Through Inconsistent Solving Perspectives

arXiv 2401.02009 ↗First posted 4 Jan 2024cs.CL cs.AIcited by 71

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

The reflection capacity of Large Language Model (LLM) has garnered extensive attention. A post-hoc prompting strategy, e.g., reflexion and self-refine, refines LLM's response based on self-evaluated or external feedback. However, recent research indicates without external feedback, LLM's intrinsic reflection is unstable. Our investigation unveils that the key bottleneck is the quality of the self-evaluated feedback. We find LLMs often exhibit overconfidence or high randomness when self-evaluate, offering stubborn or inconsistent feedback, which causes poor reflection. To remedy this, we advocate Self-Contrast: It adaptively explores diverse solving perspectives tailored to the request, contrasts the differences, and summarizes these discrepancies into a checklist which could be used to re-examine and eliminate discrepancies. Our method endows LLM with diverse perspectives to alleviate stubborn biases. Moreover, their discrepancies indicate potential errors or inherent uncertainties that LLM often overlooks. Reflecting upon these can catalyze more accurate and stable reflection. Experiments conducted on a series of reasoning and translation tasks with different LLMs serve to underscore the effectiveness and generality of our strategy.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 57 of its 83 references that are in the map. First posted on arXiv 4 Jan 2024.

Builds on

57 of 83 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Large Language Models Cannot Self-Correct Reasoning Yet3 Oct
  2. 2023Making Large Language Models Better Reasoners with Alignment5 Sep
  3. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies6 Aug
  4. 2023REFINER: Reasoning Feedback on Intermediate Representations4 Apr
  5. 2023Self-Refine: Iterative Refinement with Self-Feedback30 Mar
  6. 2022Large Language Models Can Self-Improve20 Oct
  7. 2022Language Models (Mostly) Know What They Know11 Jul

Other references · 50

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  9. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  10. 2023Let's Verify Step by Stepcited by 3,598
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  12. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
38 more
  1. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  2. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  3. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  4. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  5. 2023Teaching Large Language Models to Self-Debugcited by 963
  6. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  7. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  8. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  9. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  10. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  11. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 580
  12. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  13. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  14. 2023Language Models can Solve Computer Taskscited by 371
  15. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 356
  16. 2022Generating Sequences by Learning to Self-Correctcited by 271
  17. 2023Mathematical Capabilities of ChatGPTcited by 263
  18. 2023Self-collaboration Code Generation via ChatGPTcited by 255
  19. 2023AutoAgents: A Framework for Automatic Agent Generationcited by 211
  20. 2023LM vs LM: Detecting Factual Errors via Cross Examinationcited by 155
  21. 2023ExpertPrompting: Instructing Large Language Models to be Distinguished Expertscited by 140
  22. 2023How well do Large Language Models perform in Arithmetic tasks?cited by 130
  23. 2023Can Large Language Models Really Improve by Self-critiquing Their Own Plans?cited by 107
  24. 2023Examining Inter-Consistency of Large Language Models Collaboration: An In-depth Analysis via Debatecited by 94
  25. 2023GPT-4 Doesn't Know It's Wrong: An Analysis of Iterative Prompting for Reasoning Problemscited by 88
  26. 2023Answering Questions by Meta-Reasoning over Multiple Chains of Thoughtcited by 87
  27. 2022Solving Math Word Problems via Cooperative Reasoning induced Language Modelscited by 76
  28. 2023Data-Copilot: Bridging Billions of Data and Humans with Autonomous Workflowcited by 76
  29. 2023Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistencycited by 46
  30. 2022Learning to Reason Deductively: Math Word Problem Solving as Complex Relation Extractioncited by 44
  31. 2023Self-Polish: Enhance Reasoning in Large Language Models via Problem Refinementcited by 43
  32. 2023Automatic Model Selection with Large Language Models for Reasoningcited by 35
  33. 2023The ART of LLM Refinement: Ask, Refine, and Trustcited by 29
  34. 2022Multi-View Reasoning: Consistent Contrastive Learning for Math Word Problemcited by 13
  35. 2022A Closed-Loop Perception, Decision-Making and Reasoning Mechanism for Human-Like Navigationcited by 6
  36. 2021Learning to Navigate in a VUCA Environment: Hierarchical Multi-expert Approachcited by 5
  37. 2023Sequence-Level Certainty Reduces Hallucination In Knowledge-Grounded Dialogue Generationcited by 5
  38. 2023An Expression Tree Decoding Strategy for Mathematical Equation Generationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 30; 2025: 34; 2026: 7.

71 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  2. 2024Agent-Pro: Learning to Evolve via Policy-Level Reflection and Optimizationcited by 93
  3. 2024A Survey on Self-Evolution of Large Language Modelscited by 86
  4. 2024AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoMLcited by 84
  5. 2024Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Modelscited by 67
  6. 2024Latent Space Chain-of-Embedding Enables Output-free LLM Self-Evaluationcited by 56
  7. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 49
  8. 2024From Exploration to Mastery: Enabling LLMs to Master Tools via Self-Driven Interactionscited by 46
  9. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 43
  10. 2024Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpuscited by 32
  11. 2025Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learningcited by 32
  12. 2024Confidence v.s. Critique: A Decomposition of Self-Correction Capability for LLMscited by 28
38 more
  1. 2024Critic-CoT: Boosting the reasoning abilities of large language model via Chain-of-thoughts Criticcited by 27
  2. 2024Reinforcement Learning from Multi-role Debates as Feedback for Bias Mitigation in LLMscited by 22
  3. 2024Integrate the Essence and Eliminate the Dross: Fine-Grained Self-Consistency for Free-Form Language Generationcited by 21
  4. 2024Self-Reflection Makes Large Language Models Safer, Less Biased, and Ideologically Neutralcited by 20
  5. 2025Learning to Reason via Mixture-of-Thought for Logical Reasoningcited by 20
  6. 2025CRITICTOOL: Evaluating Self-Critique Capabilities of Large Language Models in Tool-Calling Error Scenarioscited by 19
  7. 2024Counterfactual Debating with Preset Stances for Hallucination Elimination of LLMscited by 17
  8. 2025ThinkLess: A Training-Free Inference-Efficient Method for Reducing Reasoning Redundancycited by 17
  9. 2024Mirror-Consistency: Harnessing Inconsistency in Majority Votingcited by 16
  10. 2026Auditing Multi-Agent LLM Reasoning Trees Outperforms Majority Vote and LLM-as-Judgecited by 16
  11. 2025SAMULE: Self-Learning Agents Enhanced by Multi-level Reflectioncited by 15
  12. 2024MCQG-SRefine: Multiple Choice Question Generation and Evaluation with Iterative Self-Critique, Correction, and Comparison Feedbackcited by 14
  13. 2025ProgCo: Program Helps Self-Correction of Large Language Modelscited by 14
  14. 2025Consistency in Language Models: Current Landscape, Challenges, and Future Directionscited by 14
  15. 2024CriticEval: Evaluating Large Language Model as Criticcited by 13
  16. 2024How Can LLM Guide RL? A Value-Based Approachcited by 13
  17. 2025HiFo-Prompt: Prompting with Hindsight and Foresight for LLM-based Automatic Heuristic Designcited by 13
  18. 2024Fine-Tuning on Diverse Reasoning Chains Drives Within-Inference CoT Refinement in LLMscited by 12
  19. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 12
  20. 2024RoT: Enhancing Large Language Models with Reflection on Search Treescited by 11
  21. 2024OpenEP: Open-Ended Future Event Predictioncited by 11
  22. 2024Tree-of-Code: A Tree-Structured Exploring Framework for End-to-End Code Generation and Execution in Complex Task Handlingcited by 11
  23. 2025Instruct-of-Reflection: Enhancing Large Language Models Iterative Reflection Capabilities via Dynamic-Meta Instructioncited by 10
  24. 2025AI with Emotions: Exploring Emotional Expressions in Large Language Modelscited by 10
  25. 2025Active Layer-Contrastive Decoding Reduces Hallucination in Large Language Model Generationcited by 10
  26. 2026Learning from Trials and Errors: Reflective Test-Time Planning for Embodied LLMscited by 10
  27. 2025A Probabilistic Inference Scaling Theory for LLM Self-Correctioncited by 9
  28. 2024A Survey of Calibration Process for Black-Box LLMscited by 8
  29. 2025Optimal Self-Consistency for Efficient Reasoning with Large Language Modelscited by 8
  30. 2025ToolACE-DEV: Self-Improving Tool Learning via Decomposition and EVolutioncited by 7
  31. 2024DAC: Decomposed Automation Correction for Text-to-SQLcited by 6
  32. 2025BiasFilter: An Inference-Time Debiasing Framework for Large Language Modelscited by 6
  33. 2025Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Modelscited by 6
  34. 2025Simple Yet Effective: An Information-Theoretic Approach to Multi-LLM Uncertainty Quantificationcited by 6
  35. 2024Think Thrice Before You Act: Progressive Thought Refinement in Large Language Modelscited by 5
  36. 2024EscapeBench: Towards Advancing Creative Intelligence of Language Model Agentscited by 5
  37. 2025A Review of Incorporating Psychological Theories in LLMscited by 5
  38. 2025Prompt Candidates, then Distill: A Teacher-Student Framework for LLM-driven Data Annotationcited by 5

50 most cited of 71 All 71 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024When Hindsight is Not 20/20: Testing Limits on Reflective Thinking in Large Language Models
  2. 2024Internal Consistency and Self-Feedback in Large Language Models: A Survey
  3. 2025The Self-Execution Benchmark: Measuring LLMs' Attempts to Overcome Their Lack of Self-Execution
  4. 2025Self-Evolving Critique Abilities in Large Language Models
  5. 2024Large Language Models have Intrinsic Self-Correction Ability
  6. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing
  7. 2025Self-Correcting Large Language Models: Generation vs. Multiple Choice
  8. 2025From Emergence to Control: Probing and Modulating Self-Reflection in Language Models