# Images in Discrete Choice Modeling: Addressing Data Isomorphism in Multi-Modality Inputs

arXiv: https://arxiv.org/abs/2312.14724 · HTML: https://mlsanity.com/paper/2312.14724/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 44 references that are in the map.
- First posted on arXiv 22 Dec 2023.

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [Deep Learning for Generic Object Detection: A Survey](https://mlsanity.com/paper/1809.02165/) (2018)
- [Understanding disentangling in β-VAE](https://mlsanity.com/paper/1804.03599/) (2018)
- [A Survey on the Explainability of Supervised Machine Learning](https://mlsanity.com/paper/2011.07876/) (2020)
- [Tell Me Where to Look: Guided Attention Inference Network](https://mlsanity.com/paper/1802.10171/) (2018)
- [Visual Interpretability for Deep Learning: a Survey](https://mlsanity.com/paper/1802.00614/) (2018)
- [Image inpainting: A review](https://mlsanity.com/paper/1909.06399/) (2019)
- [Choice modelling in the age of machine learning -- discussion paper](https://mlsanity.com/paper/2101.11948/) (2021)
- [A Neural-embedded Choice Model: TasteNet-MNL Modeling Taste Heterogeneity with Flexibility and Interpretability](https://mlsanity.com/paper/2002.00922/) (2020)
- [Theory-based residual neural networks: A synergy of discrete choice models and deep neural networks](https://mlsanity.com/paper/2010.11644/) (2020)
- [Assessing bikeability with street view imagery and computer vision](https://mlsanity.com/paper/2105.08499/) (2021)
- [Combining Discrete Choice Models and Neural Networks through Embeddings: Formulation, Interpretability and Performance](https://mlsanity.com/paper/2109.12042/) (2021)
- [Deep hybrid model with satellite imagery: how to combine demand modeling and computer vision for behavior analysis?](https://mlsanity.com/paper/2303.04204/) (2023)
- [A bi-partite generative model framework for analyzing and simulating large scale multiple discrete-continuous travel behaviour data](https://mlsanity.com/paper/1901.06415/) (2019)
- [Attitudes and Latent Class Choice Models using Machine learning](https://mlsanity.com/paper/2302.09871/) (2023)
- [Computer vision-enriched discrete choice models, with an application to residential location choice](https://mlsanity.com/paper/2308.08276/) (2023)

## Built on by (most cited first, 1 of 1)

- [A utility-based spatial analysis of residential street-level conditions; A case study of Rotterdam](https://mlsanity.com/paper/2410.17880/) (2024)

## Abstract (arXiv)

This paper explores the intersection of Discrete Choice Modeling (DCM) and machine learning, focusing on the integration of image data into DCM's utility functions and its impact on model interpretability. We investigate the consequences of embedding high-dimensional image data that shares isomorphic information with traditional tabular inputs within a DCM framework. Our study reveals that neural network (NN) components learn and replicate tabular variable representations from images when co-occurrences exist, thereby compromising the interpretability of DCM parameters. We propose and benchmark two methodologies to address this challenge: architectural design adjustments to segregate redundant information, and isomorphic information mitigation through source information masking and inpainting. Our experiments, conducted on a semi-synthetic dataset, demonstrate that while architectural modifications prove inconclusive, direct mitigation at the data source shows to be a more effective strategy in maintaining the integrity of DCM's interpretable parameters. The paper concludes with insights into the applicability of our findings in real-world settings and discusses the implications for future research in hybrid modeling that combines complex data modalities. Full control of tabular and image data congruence is attained by using the MIT moral machine dataset, and both inputs are merged into a choice model by deploying the Learning Multinomial Logit (L-MNL) framework.