# Delphos: A reinforcement learning framework for assisting discrete choice model specification

arXiv: https://arxiv.org/abs/2506.06410 · HTML: https://mlsanity.com/paper/2506.06410/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Interpretable and logic-guided RL, one of 15 directions in Deep reinforcement learning methods.
- Builds on 26 of its 76 references that are in the map.
- First posted on arXiv 6 Jun 2025.

## Placed in

- Deep reinforcement learning methods / [Interpretable and logic-guided RL](https://mlsanity.com/deeprl/interpretable-and-logic-guided-rl/)

## Builds on (most cited first, 26 of 26)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms](https://mlsanity.com/paper/1911.10635/) (2019)
- [Policy Distillation](https://mlsanity.com/paper/1511.06295/) (2015)
- [Designing Neural Network Architectures using Reinforcement Learning](https://mlsanity.com/paper/1611.02167/) (2016)
- [Distral: Robust Multitask Reinforcement Learning](https://mlsanity.com/paper/1707.04175/) (2017)
- [Tool Learning with Large Language Models: A Survey](https://mlsanity.com/paper/2405.17935/) (2024)
- [RUDDER: Return Decomposition for Delayed Rewards](https://mlsanity.com/paper/1806.07857/) (2018)
- [A Closer Look at Invalid Action Masking in Policy Gradient Algorithms](https://mlsanity.com/paper/2006.14171/) (2020)
- [Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods](https://mlsanity.com/paper/2404.00282/) (2024)
- [A Deeper Look at Experience Replay](https://mlsanity.com/paper/1712.01275/) (2017)
- [Massively Parallel Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1507.04296/) (2015)
- [Deep Multiagent Reinforcement Learning: Challenges and Directions](https://mlsanity.com/paper/2106.15691/) (2021)
- [Temporally-Extended ε-Greedy Exploration](https://mlsanity.com/paper/2006.01782/) (2020)
- [Choice modelling in the age of machine learning -- discussion paper](https://mlsanity.com/paper/2101.11948/) (2021)
- [Bayesian Automatic Relevance Determination for Utility Function Specification in Discrete Choice Models](https://mlsanity.com/paper/1906.03855/) (2019)
- [Elastic Step DQN: A novel multi-step algorithm to alleviate overestimation in Deep QNetworks](https://mlsanity.com/paper/2210.03325/) (2022)
- [Understanding the decision-making process of choice modellers](https://mlsanity.com/paper/2411.01704/) (2024)

## Built on by (most cited first, 2 of 2)

- [Can large language models assist choice modelling? Insights into prompting strategies and current models' capabilities](https://mlsanity.com/paper/2507.21790/) (2025)
- [Multitask Reinforcement Learning for Assisting Choice Model Specification](https://mlsanity.com/paper/2609.18441/) (2026)

## Abstract (arXiv)

We introduce Delphos, a deep reinforcement learning framework for assisting discrete choice model specification process. Delphos aims to support the modeller by providing automated, data-driven suggestions for model specifications, thereby reducing the effort required to develop and refine utility functions. Delphos conceptualises model specification as a sequential decision-making problem, inspired by the way human choice modellers iteratively construct models through a series of reasoned specification decisions. In this setting, an agent learns to specify candidate model specifications by choosing a sequence of modelling actions, such as adding alternative specific constants, accommodating both generic and alternative-specific taste parameters, applying non-linear transformations to attributes, and including interactions with covariates. Each resulting candidate model is estimated and evaluated using a reward function defined by the modeller, which can reflect statistical model fit as well as behavioural expectations. Specifically, Delphos uses a Deep Q-Network to learn how individual specification decisions contribute to the eventual quality of the resulting model and, in turn, which sequences of modelling decisions tend to produce well-performing candidates. We evaluate Delphos on both simulated and empirical datasets using alternative reward functions. In simulated cases, learning curves, Q-value patterns, and performance metrics show that Delphos learns effective specification strategies while exploring only a small fraction of the feasible modelling space. We further apply the framework to two empirical datasets to benchmark and demonstrate its practical use. These experiments illustrate the ability of Delphos to generate competitive, behaviourally plausible models and highlight the potential of this adaptive, learning-based framework to assist the model specification process.