# Creating User-steerable Projections with Interactive Semantic Mapping

arXiv: https://arxiv.org/abs/2506.15479 · HTML: https://mlsanity.com/paper/2506.15479/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 33 references that are in the map.
- First posted on arXiv 18 Jun 2025.

## Builds on (most cited first, 7 of 7)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videos](https://mlsanity.com/paper/2104.12671/) (2021)
- [A survey of dimensionality reduction techniques](https://mlsanity.com/paper/1403.2877/) (2014)
- [MoDE: CLIP Data Experts via Clustering](https://mlsanity.com/paper/2404.16030/) (2024)

## Built on by (most cited first, 3 of 3)

- [Graph Memory: A Structured and Interpretable Framework for Modality-Agnostic Embedding-Based Inference](https://mlsanity.com/paper/2511.14961/) (2025)
- [LLM-Augmented Semantic Steering of Text Embedding Projection Spaces](https://mlsanity.com/paper/2605.01957/) (2026)
- [Scalable Semantic Steering of Embedding Projections](https://mlsanity.com/paper/2607.03978/) (2026)

## Abstract (arXiv)

Dimensionality reduction (DR) techniques map high-dimensional data into lower-dimensional spaces. Yet, current DR techniques are not designed to explore semantic structure that is not directly available in the form of variables or class labels. We introduce a novel user-guided projection framework for image and text data that enables customizable, interpretable, data visualizations via zero-shot classification with Multimodal Large Language Models (MLLMs). We enable users to steer projections dynamically via natural-language guiding prompts, to specify high-level semantic relationships of interest to the users which are not explicitly present in the data dimensions. We evaluate our method across several datasets and show that it not only enhances cluster separation, but also transforms DR into an interactive, user-driven process. Our approach bridges the gap between fully automated DR techniques and human-centered data exploration, offering a flexible and adaptive way to tailor projections to specific analytical needs.