# Preference-Based Planning in Stochastic Environments: From Partially-Ordered Temporal Goals to Most Preferred Policies

arXiv: https://arxiv.org/abs/2403.18212 · HTML: https://mlsanity.com/paper/2403.18212/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 40 references that are in the map.
- First posted on arXiv 27 Mar 2024.

## Builds on (most cited first, 10 of 10)

- [A Survey of Multi-Objective Sequential Decision-Making](https://mlsanity.com/paper/1402.0590/) (2014)
- [Specifying User Preferences using Weighted Signal Temporal Logic](https://mlsanity.com/paper/2010.00752/) (2020)
- [Optimal Probabilistic Motion Planning with Potential Infeasible LTL Constraints](https://mlsanity.com/paper/2007.14325/) (2020)
- [What to Do When You Can't Do It All: Temporal Logic Planning with Soft Temporal Logic Constraints](https://mlsanity.com/paper/2008.01926/) (2020)
- [Minimum-Violation Planning for Autonomous Systems: Theoretical and Practical Considerations](https://mlsanity.com/paper/2009.11954/) (2020)
- [Probabilistic Planning with Preferences over Temporal Goals](https://mlsanity.com/paper/2103.14489/) (2021)
- [Probabilistic Planning with Partially Ordered Preferences over Temporal Goals](https://mlsanity.com/paper/2209.12267/) (2022)
- [Opportunistic Qualitative Planning in Stochastic Systems with Preferences over Temporal Logic Objectives](https://mlsanity.com/paper/2203.13803/) (2022)
- [Probabilistic Planning with Prioritized Preferences over Temporal Logic Objectives](https://mlsanity.com/paper/2304.11641/) (2023)
- [Optimal Cost-Preference Trade-off Planning with Multiple Temporal Tasks](https://mlsanity.com/paper/2306.13222/) (2023)

## Abstract (arXiv)

Human preferences are not always represented via complete linear orders: It is natural to employ partially-ordered preferences for expressing incomparable outcomes. In this work, we consider decision-making and probabilistic planning in stochastic systems modeled as Markov decision processes (MDPs), given a partially ordered preference over a set of temporally extended goals. Specifically, each temporally extended goal is expressed using a formula in Linear Temporal Logic on Finite Traces (LTLf). To plan with the partially ordered preference, we introduce order theory to map a preference over temporal goals to a preference over policies for the MDP. Accordingly, a most preferred policy under a stochastic ordering induces a stochastic nondominated probability distribution over the finite paths in the MDP. To synthesize a most preferred policy, our technical approach includes two key steps. In the first step, we develop a procedure to transform a partially ordered preference over temporal goals into a computational model, called preference automaton, which is a semi-automaton with a partial order over acceptance conditions. In the second step, we prove that finding a most preferred policy is equivalent to computing a Pareto-optimal policy in a multi-objective MDP that is constructed from the original MDP, the preference automaton, and the chosen stochastic ordering relation. Throughout the paper, we employ running examples to illustrate the proposed preference specification and solution approaches. We demonstrate the efficacy of our algorithm using these examples, providing detailed analysis, and then discuss several potential future directions.