What Drives LLM Self-Reflection? A Controlled Ablation of Uncertainty Routing in Armed Conflict Forecasting
On the map: LLM reasoning → Test-time compute and efficient thinking
Abstract (arXiv)
Self-reflection is widely assumed to improve LLM reasoning, yet which component drives the gain remains poorly understood. We present a controlled six-condition ablation isolating four components of LLM self-reflection: evidence exposure, diagnostic scaffolding, taxonomy vocabulary, and action routing. Two precise null results converge on a single mechanism. First, structured diagnostic questions add no measurable value over unstructured reflection (F1 = 0.296 vs 0.297, p = 1.000, 95% CI [-0.041, +0.040]). Second, presenting the full uncertainty taxonomy while collapsing the action space to a single generic action also adds no value (DeltaF1 = +0.008, overlapping 95% CIs), ruling out taxonomy vocabulary as the mechanism. Typed action routing provides consistent directional gains (F1 = 0.379 vs 0.296); the conservative estimate controlling for taxonomy vocabulary is DeltaF1 = +0.075, and the overall gain over the single-shot baseline is significant by bootstrap CI (DeltaF1 = +0.101, 95% CI [+0.020, +0.185]). The vocabulary-routing decomposition replicates on GPT-4o: taxonomy vocabulary adds no significant value over generic reflection (p = 0.773), while action routing provides significant gains (p = 0.025), confirming the mechanism holds across backbones. Gains concentrate on structurally novel conflicts: in Myanmar (F1: 0.000 rightarrow 0.353) and Ukraine (0.167 rightarrow 0.500), the vocabulary-only condition recovers no more than generic reflection while action routing breaks the degenerate prior. These findings identify typed action routing -- not diagnostic scaffolding or taxonomy vocabulary -- as a promising design principle for metacognitive LLM forecasting agents, while motivating larger-scale evaluation across conflict typologies.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 10 of its 24 references that are in the map. First posted on arXiv 29 May 2026.
Builds on
10 of 24 references.
Closest ancestors · 1
Same direction, newest first
Other references · 9
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2017On Calibration of Modern Neural Networkscited by 4,960
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
- 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,011
- 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
- 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 913
- 2023Detecting Human Rights Violations on Social Media during Russia-Ukraine Warcited by 3
- 2025Do Large Language Models Know Conflict? Investigating Parametric vs. Non-Parametric Knowledge of LLMs for Conflict Forecastingcited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025ReflCtrl: Controlling LLM Reflection Efficiently via Representation Engineering
- 2026Understanding Reasoning in LLMs through Strategic Information Allocation under Uncertainty
- 2026From Plausible to Actionable: A Position on LLM Self-Explanations
- 2026ReFlect: An Effective Harness System for Complex Long-Horizon LLM Reasoning
- 2025Finding the Sweet Spot: Trading Quality, Cost, and Speed During Inference-Time LLM Reflection
- 2024Self-Contrast: Better Reflection Through Inconsistent Solving Perspectives
- 2024Self-Reflection Makes Large Language Models Safer, Less Biased, and Ideologically Neutral
- 2024When Hindsight is Not 20/20: Testing Limits on Reflective Thinking in Large Language Models