# DecisionFlow: Advancing Large Language Model as Principled Decision Maker

arXiv: https://arxiv.org/abs/2505.21397 · HTML: https://mlsanity.com/paper/2505.21397/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 25 references that are in the map.
- First posted on arXiv 27 May 2025.

## Builds on (most cited first, 10 of 10)

- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://mlsanity.com/paper/2211.12588/) (2022)
- [Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP](https://mlsanity.com/paper/2212.14024/) (2022)
- [Foundation Models for Decision Making: Problems, Methods, and Opportunities](https://mlsanity.com/paper/2303.04129/) (2023)
- [A Language Agent for Autonomous Driving](https://mlsanity.com/paper/2311.10813/) (2023)
- [Fact-Checking Complex Claims with Program-Guided Reasoning](https://mlsanity.com/paper/2305.12744/) (2023)
- [Faithful Logical Reasoning via Symbolic Chain-of-Thought](https://mlsanity.com/paper/2405.18357/) (2024)
- [Determinants of LLM-assisted Decision-Making](https://mlsanity.com/paper/2402.17385/) (2024)
- [The Importance of Directional Feedback for LLM-based Optimizers](https://mlsanity.com/paper/2405.16434/) (2024)
- [Language Models are Alignable Decision-Makers: Dataset and Application to the Medical Triage Domain](https://mlsanity.com/paper/2406.06435/) (2024)

## Built on by (most cited first, 12 of 12)

- [Value of Information: A Framework for Human-Agent Communication](https://mlsanity.com/paper/2601.06407/) (2026)
- [Auto-Dreamer: Learning Offline Memory Consolidation for Language Agents](https://mlsanity.com/paper/2605.20616/) (2026)
- [Guiding LLM Decision-Making with Fairness Reward Models](https://mlsanity.com/paper/2507.11344/) (2025)
- [Building Social World Models with Large Language Models](https://mlsanity.com/paper/2606.11482/) (2026)
- [Meet Dynamic Individual Preferences: Resolving Conflicting Human Value with Paired Fine-Tuning](https://mlsanity.com/paper/2604.12479/) (2026)
- [LiveSearchBench: An Automatically Constructed Benchmark for Retrieval and Reasoning over Dynamic Knowledge](https://mlsanity.com/paper/2511.01409/) (2025)
- [Momentum-Coupled Rubric Adaptation for Detailed Image Captioning](https://mlsanity.com/paper/2609.36893/) (2026)
- [Classical AI vs. LLMs for Decision-Maker Alignment in Health Insurance Choices](https://mlsanity.com/paper/2510.06093/) (2025)
- [CONDESION-BENCH: Conditional Decision-Making of Large Language Models in Compositional Action Space](https://mlsanity.com/paper/2604.09029/) (2026)
- [Decisive: Guiding User Decisions with Optimal Preference Elicitation from Unstructured Documents](https://mlsanity.com/paper/2604.18122/) (2026)
- [Harness-Search: Guiding Long-Horizon Search through Multi-Agent Coordination](https://mlsanity.com/paper/2610.05382/) (2026)
- [Learning to Simulate Individuals from Macro Social Signals](https://mlsanity.com/paper/2610.07062/) (2026)

## Abstract (arXiv)

In high-stakes domains such as healthcare and finance, effective decision-making demands not just accurate outcomes but transparent and explainable reasoning. However, current language models often lack the structured deliberation needed for such tasks, instead generating decisions and justifications in a disconnected, post-hoc manner. To address this, we propose DecisionFlow, a novel decision modeling framework that guides models to reason over structured representations of actions, attributes, and constraints. Rather than predicting answers directly from prompts, DecisionFlow builds a semantically grounded decision space and infers a latent utility function to evaluate trade-offs in a transparent, utility-driven manner. This process produces decisions tightly coupled with interpretable rationales reflecting the model's reasoning. Empirical results on two high-stakes benchmarks show that DecisionFlow not only achieves up to 30% accuracy gains over strong prompting baselines but also enhances alignment in outcomes. Our work is a critical step toward integrating symbolic reasoning with LLMs, enabling more accountable, explainable, and reliable LLM decision support systems. Code and data are at https://github.com/xiusic/DecisionFlow.