# DeepSI: Interactive Deep Learning for Semantic Interaction

arXiv: https://arxiv.org/abs/2305.18357 · HTML: https://mlsanity.com/paper/2305.18357/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 69 references that are in the map.
- First posted on arXiv 26 May 2023.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [Universal Sentence Encoder](https://mlsanity.com/paper/1803.11175/) (2018)
- [Deep metric learning using Triplet network](https://mlsanity.com/paper/1412.6622/) (2014)
- [To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks](https://mlsanity.com/paper/1903.05987/) (2019)
- [Dive into Deep Learning](https://mlsanity.com/paper/2106.11342/) (2021)
- [Deep Metric Learning with Angular Loss](https://mlsanity.com/paper/1708.01682/) (2017)
- [RetainVis: Visual Analytics with Interpretable and Interactive Recurrent Neural Networks on Electronic Medical Records](https://mlsanity.com/paper/1805.10724/) (2018)
- [Visual Interaction with Deep Learning Models through Collaborative Semantic Inference](https://mlsanity.com/paper/1907.10739/) (2019)
- [Deep Learning Multidimensional Projections](https://mlsanity.com/paper/1902.07958/) (2019)
- [Evaluation of Interactive Machine Learning Systems](https://mlsanity.com/paper/1801.07964/) (2018)
- [DeepVA: Bridging Cognition and Computation through Semantic Interaction and Deep Learning](https://mlsanity.com/paper/2007.15800/) (2020)
- [Evaluating Semantic Interaction on Word Embeddings via Simulation](https://mlsanity.com/paper/2007.15824/) (2020)
- [Sharkzor: Interactive Deep Learning for Image Triage, Sort and Summary](https://mlsanity.com/paper/1802.05316/) (2018)

## Built on by (most cited first, 7 of 7)

- [Multi-objective Deep Learning: Taxonomy and Survey of the State of the Art](https://mlsanity.com/paper/2412.01566/) (2024)
- [Interactive Narrative Analytics: Bridging Computational Narrative Extraction and Human Sensemaking](https://mlsanity.com/paper/2601.11459/) (2026)
- [ThematicPlane: Bridging Tacit User Intent and Latent Spaces for Image Generation](https://mlsanity.com/paper/2508.06065/) (2025)
- [LLM-Augmented Semantic Steering of Text Embedding Projection Spaces](https://mlsanity.com/paper/2605.01957/) (2026)
- [Semantic Prompting: Agentic Incremental Narrative Refinement through Spatial Semantic Interaction](https://mlsanity.com/paper/2604.19971/) (2026)
- [Scalable Semantic Steering of Embedding Projections](https://mlsanity.com/paper/2607.03978/) (2026)
- [KeySI: An Interaction Framework for Tuning Text Embeddings Based on Human Feedback](https://mlsanity.com/paper/2607.20556/) (2026)

## Abstract (arXiv)

In this paper, we design novel interactive deep learning methods to improve semantic interactions in visual analytics applications. The ability of semantic interaction to infer analysts' precise intents during sensemaking is dependent on the quality of the underlying data representation. We propose the DeepSIfinetune framework that integrates deep learning into the human-in-the-loop interactive sensemaking pipeline, with two important properties. First, deep learning extracts meaningful representations from raw data, which improves semantic interaction inference. Second, semantic interactions are exploited to fine-tune the deep learning representations, which then further improves semantic interaction inference. This feedback loop between human interaction and deep learning enables efficient learning of user- and task-specific representations. To evaluate the advantage of embedding the deep learning within the semantic interaction loop, we compare DeepSIfinetune against a state-of-the-art but more basic use of deep learning as only a feature extractor pre-processed outside of the interactive loop. Results of two complementary studies, a human-centered qualitative case study and an algorithm-centered simulation-based quantitative experiment, show that DeepSIfinetune more accurately captures users' complex mental models with fewer interactions.