# Controlled Paraphrase Geometry in Sentence Embedding Space: Local Manifold Modeling and Latent Probing

arXiv: https://arxiv.org/abs/2605.01073 · HTML: https://mlsanity.com/paper/2605.01073/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 31 references that are in the map.
- First posted on arXiv 1 May 2026.

## Builds on (most cited first, 20 of 20)

- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations](https://mlsanity.com/paper/1805.06201/) (2018)
- [MixUp as Locally Linear Out-Of-Manifold Regularization](https://mlsanity.com/paper/1809.02499/) (2018)
- [Dataset Augmentation in Feature Space](https://mlsanity.com/paper/1702.05538/) (2017)
- [A Survey on Data Augmentation for Text Classification](https://mlsanity.com/paper/2107.03158/) (2021)
- [Parametric UMAP embeddings for representation and semi-supervised learning](https://mlsanity.com/paper/2009.12981/) (2020)
- [How Effective is Task-Agnostic Data Augmentation for Pretrained Transformers?](https://mlsanity.com/paper/2010.01764/) (2020)
- [Multi-SimLex: A Large-Scale Evaluation of Multilingual and Cross-Lingual Lexical Semantic Similarity](https://mlsanity.com/paper/2003.04866/) (2020)
- [Still a Pain in the Neck: Evaluating Text Representations on Lexical Composition](https://mlsanity.com/paper/1902.10618/) (2019)
- [Holmes: A Benchmark to Assess the Linguistic Competence of Language Models](https://mlsanity.com/paper/2404.18923/) (2024)
- [On-manifold Adversarial Data Augmentation Improves Uncertainty Calibration](https://mlsanity.com/paper/1912.07458/) (2019)
- [Sentence Similarity Based on Contexts](https://mlsanity.com/paper/2105.07623/) (2021)
- [TextManiA: Enriching Visual Feature by Text-driven Manifold Augmentation](https://mlsanity.com/paper/2307.14611/) (2023)
- [Collective Human Opinions in Semantic Textual Similarity](https://mlsanity.com/paper/2308.04114/) (2023)
- [Data Augmentation with Manifold Exploring Geometric Transformations for Increased Performance and Robustness](https://mlsanity.com/paper/1901.04420/) (2019)
- [The Impact of Word Splitting on the Semantic Content of Contextualized Word Representations](https://mlsanity.com/paper/2402.14616/) (2024)
- [Augment on Manifold: Mixup Regularization with UMAP](https://mlsanity.com/paper/2312.13141/) (2023)

## Abstract (arXiv)

The paper studies the local geometry of embedding clouds induced by controlled local classes of semantically close sentences. The central question is how controlled paraphrase-like semantic variation is organized in sentence embedding space and whether this local structure can be explicitly modeled by low-degree fitted carriers. We introduce a local geometric modeling scheme based on affine, quadratic, and cubic fitted models. We also use a surface-based latent probing procedure that constructs synthetic latent points in a reduced local PCA space with respect to the fitted carrier. The procedure is intended as an offline method for representation-space analysis, local manifold modeling, and geometry-aware latent probing. Generated latent points are evaluated using criteria that measure consistency with the fitted surface, preservation of neighborhood structure, agreement with the empirical distribution, stability of Hessian-based second-order shape descriptors, and stability of fitted-model coefficients. Experiments on controlled sets of semantically close sentences show that nonlinear local models describe embedding clouds more accurately than affine models. Surface-based generation provides strong fitted-geometry fidelity, including surface consistency, Hessian-based shape consistency, and coefficient consistency. Downstream experiments show that geometric validity of synthetic latent points does not automatically translate into improved classification performance. The results support explicit local geometric modeling of sentence embedding space and highlight the need to distinguish geometric validity from discriminative utility. As a resource contribution, we introduce CoPaGE-300K, a controlled template-based dataset of semantically close sentence variants with slot-level annotations and precomputed sentence embeddings.