# Scalable Semantic Steering of Embedding Projections

arXiv: https://arxiv.org/abs/2607.03978 · HTML: https://mlsanity.com/paper/2607.03978/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 28 references that are in the map.
- First posted on arXiv 4 Jul 2026.

## Builds on (most cited first, 16 of 16)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [What does a platypus look like? Generating customized prompts for zero-shot image classification](https://mlsanity.com/paper/2209.03320/) (2022)
- [Multimodal Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/2011.08899/) (2020)
- [Multi-label classification for biomedical literature: an overview of the BioCreative VII LitCovid Track for COVID-19 literature topic annotations](https://mlsanity.com/paper/2204.09781/) (2022)
- [Proto-CLIP: Vision-Language Prototypical Network for Few-Shot Learning](https://mlsanity.com/paper/2307.03073/) (2023)
- [Class-constrained t-SNE: Combining Data Features and Class Probabilities](https://mlsanity.com/paper/2308.13837/) (2023)
- [DeepSI: Interactive Deep Learning for Semantic Interaction](https://mlsanity.com/paper/2305.18357/) (2023)
- [ModalChorus: Visual Probing and Alignment of Multi-modal Embeddings via Modal Fusion Map](https://mlsanity.com/paper/2407.12315/) (2024)
- [ImageSI: Semantic Interaction for Deep Learning Image Projections](https://mlsanity.com/paper/2408.03845/) (2024)
- [Creating User-steerable Projections with Interactive Semantic Mapping](https://mlsanity.com/paper/2506.15479/) (2025)
- [Empowering Sentence Encoders with Prompting and Label Retrieval for Zero-shot Text Classification](https://mlsanity.com/paper/2212.10391/) (2022)
- [Visualizing Spatial Semantics of Dimensionally Reduced Text Embeddings](https://mlsanity.com/paper/2409.03949/) (2024)
- [Cross-Modal Prototype Alignment and Mixing for Training-Free Few-Shot Classification](https://mlsanity.com/paper/2603.24528/) (2026)
- [LangLasso: Interactive Cluster Descriptions through LLM Explanation](https://mlsanity.com/paper/2601.10458/) (2026)
- [LLM-Augmented Semantic Steering of Text Embedding Projection Spaces](https://mlsanity.com/paper/2605.01957/) (2026)

## Abstract (arXiv)

Low-dimensional projections support interactive visual analysis of high-dimensional data embeddings, but their structure often does not align with analyst-defined semantic relationships. Recent LLM-augmented semantic steering methods address this gap by externalizing analyst intent from user-defined groups of seed examples, but they propagate intent through per-item LLM reasoning, causing LLM calls and cost to grow linearly with collection size. We propose a scalable semantic steering method that shifts semantic computation from individual items to user-defined groups. A single LLM call generates structured profiles for all groups, which are embedded and combined with seed centroids to form hybrid semantic prototypes. The method then propagates intent without retraining, using embedding-space soft assignment, abstention, and alignment-scaled updates before reprojection. On a 5K-document LitCovid corpus, our method achieves global alignment comparable to per-item LLM steering while reducing LLM calls by over three orders of magnitude. An image case study shows that the same prototype-based mechanism extends to multimodal embeddings. These results suggest that group-level representations can make semantic steering more practical for larger embedding collections.