# Multitask Reinforcement Learning for Assisting Choice Model Specification

arXiv: https://arxiv.org/abs/2609.18441 · HTML: https://mlsanity.com/paper/2609.18441/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 11 of its 38 references that are in the map.
- First posted on arXiv 16 Sep 2026.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 11 of 11)

- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Multi-Task Reinforcement Learning with Context-based Representations](https://mlsanity.com/paper/2102.06177/) (2021)
- [A Closer Look at Invalid Action Masking in Policy Gradient Algorithms](https://mlsanity.com/paper/2006.14171/) (2020)
- [Multi-task Deep Reinforcement Learning with PopArt](https://mlsanity.com/paper/1809.04474/) (2018)
- [Choice modelling in the age of machine learning -- discussion paper](https://mlsanity.com/paper/2101.11948/) (2021)
- [Dynamic Interaction-Aware Scene Understanding for Reinforcement Learning in Autonomous Driving](https://mlsanity.com/paper/1909.13582/) (2019)
- [Bayesian Automatic Relevance Determination for Utility Function Specification in Discrete Choice Models](https://mlsanity.com/paper/1906.03855/) (2019)
- [Understanding the decision-making process of choice modellers](https://mlsanity.com/paper/2411.01704/) (2024)
- [Delphos: A reinforcement learning framework for assisting discrete choice model specification](https://mlsanity.com/paper/2506.06410/) (2025)
- [Can large language models assist choice modelling? Insights into prompting strategies and current models' capabilities](https://mlsanity.com/paper/2507.21790/) (2025)

## Abstract (arXiv)

Discrete choice model specification is a time-consuming task in which modellers often specify and estimate multiple models while balancing goodness-of-fit, parsimony, and behavioural plausibility. We present Delphos, a multitask reinforcement learning framework that learns transferable specification strategies across transport choice datasets. Delphos frames model specification as a sequential decision-making problem in which it applies a sequence of modelling actions and receives feedback from an estimation environment based on model performance and convergence. To transfer modelling decisions across datasets with different sets of variables, Delphos represents utility specifications as sets of modelling terms using a DeepSet-Q architecture, allowing a shared specification policy to learn across multiple datasets. Trained on nine transport choice datasets, Delphos consistently outperforms independently trained single-task agents, indicating that sharing modelling experience improves learning efficiency and helps identify promising sequences of modelling decisions with fewer unsuccessful estimation attempts. When applied without further training to the unseen Swissmetro and Decisions datasets, the same agent identifies competitive specifications in less than 20 minutes on a standard CPU. It achieves a higher log-likelihood per observation than the VNS metaheuristic on Swissmetro and performance comparable to a published MNL specification developed by expert modellers on Decisions. These findings show that accumulating and reusing modelling experience enables Delphos to function as an intelligent assistant for discrete choice model specification. It reduces manual trial-and-error while allowing modellers to retain control over model diagnosis, refinement, and final selection.