# Visual Interaction with Deep Learning Models through Collaborative Semantic Inference

arXiv: https://arxiv.org/abs/1907.10739 · HTML: https://mlsanity.com/paper/1907.10739/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jul 2019.

## Built on by (most cited first, 24 of 24)

- [The Impact of Multiple Parallel Phrase Suggestions on Email Input and Composition Behaviour of Native and Non-Native English Writers](https://mlsanity.com/paper/2101.09157/) (2021)
- [UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobility](https://mlsanity.com/paper/2501.02341/) (2025)
- [Is Sparse Attention more Interpretable?](https://mlsanity.com/paper/2106.01087/) (2021)
- [Sequence-to-Sequence Learning with Latent Neural Grammars](https://mlsanity.com/paper/2109.01135/) (2021)
- [The State of the Art in Enhancing Trust in Machine Learning Models with the Use of Visualizations](https://mlsanity.com/paper/2212.11737/) (2022)
- [Visual Analytics For Machine Learning: A Data Perspective Survey](https://mlsanity.com/paper/2307.07712/) (2023)
- [SynthBio: A Case Study in Human-AI Collaborative Curation of Text Datasets](https://mlsanity.com/paper/2111.06467/) (2021)
- [A Unified Understanding of Deep NLP Models for Text Classification](https://mlsanity.com/paper/2206.09355/) (2022)
- [Diagnosing AI Explanation Methods with Folk Concepts of Behavior](https://mlsanity.com/paper/2201.11239/) (2022)
- [Towards Explainable Evaluation Metrics for Machine Translation](https://mlsanity.com/paper/2306.13041/) (2023)
- [Harnessing the Power of LLMs: Evaluating Human-AI Text Co-Creation through the Lens of News Headline Generation](https://mlsanity.com/paper/2310.10706/) (2023)
- [SummVis: Interactive Visual Analysis of Models, Data, and Evaluation for Text Summarization](https://mlsanity.com/paper/2104.07605/) (2021)
- [Nine Potential Pitfalls when Designing Human-AI Co-Creative Systems](https://mlsanity.com/paper/2104.00358/) (2021)
- [GenNI: Human-AI Collaboration for Data-Backed Text Generation](https://mlsanity.com/paper/2110.10185/) (2021)
- [Text-Blueprint: An Interactive Platform for Plan-based Conditional Generation](https://mlsanity.com/paper/2305.00034/) (2023)
- [DeepSI: Interactive Deep Learning for Semantic Interaction](https://mlsanity.com/paper/2305.18357/) (2023)
- [USB: A Unified Summarization Benchmark Across Tasks and Domains](https://mlsanity.com/paper/2305.14296/) (2023)
- [generAItor: Tree-in-the-Loop Text Generation for Language Model Explainability and Adaptation](https://mlsanity.com/paper/2403.07627/) (2024)
- [Active learning for medical code assignment](https://mlsanity.com/paper/2104.05741/) (2021)
- [Bengali Handwritten Digit Recognition using CNN with Explainable AI](https://mlsanity.com/paper/2212.12146/) (2022)
- [SpaceEditing: Integrating Human Knowledge into Deep Neural Networks via Interactive Latent Space Editing](https://mlsanity.com/paper/2212.04065/) (2022)
- [LLM-Augmented Semantic Steering of Text Embedding Projection Spaces](https://mlsanity.com/paper/2605.01957/) (2026)
- [Eliciting Model Steering Interactions from Users via Data and Visual Design Probes](https://mlsanity.com/paper/2310.09314/) (2023)
- [Explanation User Interfaces: A Systematic Literature Review](https://mlsanity.com/paper/2505.20085/) (2025)

## Abstract (arXiv)

Automation of tasks can have critical consequences when humans lose agency over decision processes. Deep learning models are particularly susceptible since current black-box approaches lack explainable reasoning. We argue that both the visual interface and model structure of deep learning systems need to take into account interaction design. We propose a framework of collaborative semantic inference (CSI) for the co-design of interactions and models to enable visual collaboration between humans and algorithms. The approach exposes the intermediate reasoning process of models which allows semantic interactions with the visual metaphors of a problem, which means that a user can both understand and control parts of the model reasoning process. We demonstrate the feasibility of CSI with a co-designed case study of a document summarization system.