Reproducing and Stress-Testing Two Approaches to LLM Reasoning Reliability: Test-Time Probability Aggregation and Logic-Representation Editing
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
We independently reproduce two recent methods for making large language model (LLM) reasoning more reliable, and stress-test them across domains and models (RPC across four new task domains with Qwen3-8B, LCF across four 7-8B models). The first, RPC, aggregates token probabilities and self-consistency at inference; the second, LCF, trains projectors that split hidden states into "content" and "logic" and edits the logic part toward a valid region. Validating such reliability claims matters because the original evaluations are run by each method's own authors and were never independently reproduced or stress-tested across models and domains, and LCF shipped no public code. We re-run RPC's published-path aggregation and re-implement LCF's projector, contrastive, and intervention pipeline, then extend both to text-to-SQL, legal extraction, fallacy identification, and precedent grading, and probe LCF's representation directly. RPC reproduces the original grid exactly on the authors' released reasoning paths; on four new domains its edge over self-consistency is never significant (ties or small mixed differences, paired p >= 0.28), and on BIRD, the one domain where we vary the budget, the edge grows with K as predicted but its largest gap (+2.5 accuracy at K=32, p=0.16) reverses to -0.25 when we enlarge the sample to n=200. LCF's logic-validity direction is real but weak (0.82 separability at the single best sub-layer versus 0.95 for a semantic-attribute control); its one positive effect (Qwen3 DeltaProb) is not significant (p=0.56), while it significantly reduces DeltaProb on two of the other three models.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 14 of its 47 references that are in the map. First posted on arXiv 9 Aug 2026.
Builds on
14 of 47 references.
Closest ancestors · 3
Same direction, newest first
- 2025Qwen3 Technical Report14 May
- 2024Reasoning Circuits in Language Models: A Mechanistic Interpretation of Syllogistic Inference16 Aug
- 2022Language models show human-like content effects on reasoning tasks14 Jul
Other references · 11
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2023Mistral 7Bcited by 3,065
- 2022Language Models (Mostly) Know What They Knowcited by 1,783
- 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
- 2023Steering Language Models With Activation Engineeringcited by 824
- 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
- 2024A Looming Replication Crisis in Evaluating Behavior in Language Models? Evidence and Solutionscited by 8
- 2025How Language Models Conflate Logical Validity with Plausibility: A Representational Analysis of Content Effectscited by 3
- 2026Reasoning or Fluency? Dissecting Probabilistic Confidence in Best-of-N Selectioncited by 2
- 2026Decodable but Not Corrected by Fixed Residual-Stream Linear Steering: Evidence from Medical LLM Failure Regimescited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMs
- 2025LLMSR@XLLM25: An Empirical Study of LLM for Structural Reasoning
- 2024LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models
- 2026Stress-Testing the Reasoning Competence of LLMs With Proofs Under Minimal Formalism
- 2026LGMT: Logic-Grounded Metamorphic Testing for Evaluating the Reasoning Reliability of LLMs
- 2025A Theoretical Study on Bridging Internal Probability and Self-Consistency for LLM Reasoning
- 2025Less is More: Improving LLM Reasoning with Minimal Test-Time Intervention
- 2025Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs