# LLM-ARC: Enhancing LLMs with an Automated Reasoning Critic

arXiv: https://arxiv.org/abs/2406.17663 · HTML: https://mlsanity.com/paper/2406.17663/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 0 of its 19 references that are in the map.
- First posted on arXiv 25 Jun 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Built on by (most cited first, 16 of 16)

- [What's Wrong? Refining Meeting Summaries with LLM Feedback](https://mlsanity.com/paper/2407.11919/) (2024)
- [Instantiation-based Formalization of Logical Reasoning Tasks using Language Models and Logical Solvers](https://mlsanity.com/paper/2501.16961/) (2025)
- [From Reasoning to Learning: A Survey on Hypothesis Discovery and Rule Learning with Large Language Models](https://mlsanity.com/paper/2505.21935/) (2025)
- [Mind Scramble: Unveiling Large Language Model Psychology Via Typoglycemia](https://mlsanity.com/paper/2410.01677/) (2024)
- [Integrating Expert Knowledge into Logical Programs via LLMs](https://mlsanity.com/paper/2502.12275/) (2025)
- [SR-LLM: Rethinking the Structured Representation in Large Language Model](https://mlsanity.com/paper/2502.14352/) (2025)
- [Building Data-Driven Occupation Taxonomies: A Bottom-Up Multi-Stage Approach via Semantic Clustering and Multi-Agent Collaboration](https://mlsanity.com/paper/2509.15786/) (2025)
- [Hypothesis Generation via LLM-Automated Language Bias for ILP](https://mlsanity.com/paper/2505.21486/) (2025)
- [GAIA: A Data Flywheel System for Training GUI Test-Time Scaling Critic Models](https://mlsanity.com/paper/2601.18197/) (2026)
- [VeriSimpl: Robust Optimization Modeling from Natural Language using Simplification-based Verification](https://mlsanity.com/paper/2607.20474/) (2026)
- [Artificial Human Intelligence: The role of Humans in the Development of Next Generation AI](https://mlsanity.com/paper/2409.16001/) (2024)
- [Synergizing LLMs and Knowledge Graphs: A Novel Approach to Software Repository-Related Question Answering](https://mlsanity.com/paper/2412.03815/) (2024)
- [An LLM + ASP Workflow for Joint Entity-Relation Extraction](https://mlsanity.com/paper/2508.12611/) (2025)
- [Neurosymbolic Language Reasoning as Satisfiability Modulo Theory](https://mlsanity.com/paper/2602.18095/) (2026)
- [Accelerating NeurASP with vectorization and caching](https://mlsanity.com/paper/2606.10787/) (2026)
- [Automatic Conversion of NICE Guidelines to an Executable Computational Model Using Large Language Models](https://mlsanity.com/paper/2608.30022/) (2026)

## Abstract (arXiv)

We introduce LLM-ARC, a neuro-symbolic framework designed to enhance the logical reasoning capabilities of Large Language Models (LLMs), by combining them with an Automated Reasoning Critic (ARC). LLM-ARC employs an Actor-Critic method where the LLM Actor generates declarative logic programs along with tests for semantic correctness, while the Automated Reasoning Critic evaluates the code, runs the tests and provides feedback on test failures for iterative refinement. Implemented using Answer Set Programming (ASP), LLM-ARC achieves a new state-of-the-art accuracy of 88.32% on the FOLIO benchmark which tests complex logical reasoning capabilities. Our experiments demonstrate significant improvements over LLM-only baselines, highlighting the importance of logic test generation and iterative self-refinement. We achieve our best result using a fully automated self-supervised training loop where the Actor is trained on end-to-end dialog traces with Critic feedback. We discuss potential enhancements and provide a detailed error analysis, showcasing the robustness and efficacy of LLM-ARC for complex natural language reasoning tasks.