# The Effect of Document Selection on Query-focused Text Analysis

arXiv: https://arxiv.org/abs/2604.12099 · HTML: https://mlsanity.com/paper/2604.12099/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 51 references that are in the map.
- First posted on arXiv 13 Apr 2026.

## Builds on (most cited first, 10 of 10)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [Passage Re-ranking with BERT](https://mlsanity.com/paper/1901.04085/) (2019)
- [Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds](https://mlsanity.com/paper/1906.03671/) (2019)
- [Autoencoding Variational Inference For Topic Models](https://mlsanity.com/paper/1703.01488/) (2017)
- [Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooM](https://mlsanity.com/paper/2404.12259/) (2024)
- [Out-of-Domain Semantics to the Rescue! Zero-Shot Hybrid Retrieval Models](https://mlsanity.com/paper/2201.10582/) (2022)
- [Improving the TENOR of Labeling: Re-evaluating Topic Models for Content Analysis](https://mlsanity.com/paper/2401.16348/) (2024)
- [Retrieval Augmented Generation for Topic Modeling in Organizational Research: An Introduction with Empirical Demonstration](https://mlsanity.com/paper/2502.20963/) (2025)
- [Mapping Patient-Perceived Physician Traits from Nationwide Online Reviews with LLMs](https://mlsanity.com/paper/2510.03997/) (2025)

## Abstract (arXiv)

Analyses of document collections often require selecting what data to analyze, as not all documents are relevant to a particular research question and computational constraints preclude analyzing all documents, yet little work has examined effects of selection strategy choices. We systematically evaluate seven selection methods (from random selection to hybrid retrieval) on outputs from four text analyses methods (LDA, BERTopic, TopicGPT, HiCode) over two datasets with 26 open-ended queries. Our evaluation reveals practice guidance: semantic or hybrid retrieval offer strong go-to approaches that avoid the pitfalls of weaker selection strategies and the unnecessary compute overhead of more complicated ones. Overall, our evaluation framework establishes data selection as a methodological decision, rather than a practical necessity, inviting the development of new strategies.