# What Drives LLM Self-Reflection? A Controlled Ablation of Uncertainty Routing in Armed Conflict Forecasting

arXiv: https://arxiv.org/abs/2608.12322 · HTML: https://mlsanity.com/paper/2608.12322/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 10 of its 24 references that are in the map.
- First posted on arXiv 29 May 2026.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 10 of 10)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate](https://mlsanity.com/paper/2305.19118/) (2023)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [Detecting Human Rights Violations on Social Media during Russia-Ukraine War](https://mlsanity.com/paper/2306.05370/) (2023)
- [Do Large Language Models Know Conflict? Investigating Parametric vs. Non-Parametric Knowledge of LLMs for Conflict Forecasting](https://mlsanity.com/paper/2505.09852/) (2025)

## Abstract (arXiv)

Self-reflection is widely assumed to improve LLM reasoning, yet which component drives the gain remains poorly understood. We present a controlled six-condition ablation isolating four components of LLM self-reflection: evidence exposure, diagnostic scaffolding, taxonomy vocabulary, and action routing. Two precise null results converge on a single mechanism. First, structured diagnostic questions add no measurable value over unstructured reflection (F1 = 0.296 vs 0.297, p = 1.000, 95% CI [-0.041, +0.040]). Second, presenting the full uncertainty taxonomy while collapsing the action space to a single generic action also adds no value (DeltaF1 = +0.008, overlapping 95% CIs), ruling out taxonomy vocabulary as the mechanism. Typed action routing provides consistent directional gains (F1 = 0.379 vs 0.296); the conservative estimate controlling for taxonomy vocabulary is DeltaF1 = +0.075, and the overall gain over the single-shot baseline is significant by bootstrap CI (DeltaF1 = +0.101, 95% CI [+0.020, +0.185]). The vocabulary-routing decomposition replicates on GPT-4o: taxonomy vocabulary adds no significant value over generic reflection (p = 0.773), while action routing provides significant gains (p = 0.025), confirming the mechanism holds across backbones. Gains concentrate on structurally novel conflicts: in Myanmar (F1: 0.000 rightarrow 0.353) and Ukraine (0.167 rightarrow 0.500), the vocabulary-only condition recovers no more than generic reflection while action routing breaks the degenerate prior. These findings identify typed action routing -- not diagnostic scaffolding or taxonomy vocabulary -- as a promising design principle for metacognitive LLM forecasting agents, while motivating larger-scale evaluation across conflict typologies.