# An Interdisciplinary Perspective on Evaluation and Experimental Design for Visual Text Analytics: Position Paper

arXiv: https://arxiv.org/abs/2209.11534 · HTML: https://mlsanity.com/paper/2209.11534/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 104 references that are in the map.
- First posted on arXiv 23 Sep 2022.

## Builds on (most cited first, 15 of 15)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [A Survey of Data Augmentation Approaches for NLP](https://mlsanity.com/paper/2105.03075/) (2021)
- [Calibration of Pre-trained Transformers](https://mlsanity.com/paper/2003.07892/) (2020)
- [What Will it Take to Fix Benchmarking in Natural Language Understanding?](https://mlsanity.com/paper/2104.02145/) (2021)
- [A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores](https://mlsanity.com/paper/2002.08035/) (2020)
- [Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing](https://mlsanity.com/paper/1807.00914/) (2018)
- [Targeting the Benchmark: On Methodology in Current Natural Language Processing Research](https://mlsanity.com/paper/2007.04792/) (2020)
- [Utility is in the Eye of the User: A Critique of NLP Leaderboards](https://mlsanity.com/paper/2009.13888/) (2020)
- [Calibrated Language Model Fine-Tuning for In- and Out-of-Distribution Data](https://mlsanity.com/paper/2010.11506/) (2020)
- [X-SRL: A Parallel Cross-Lingual Semantic Role Labeling Dataset](https://mlsanity.com/paper/2010.01998/) (2020)
- [Cartolabe: A Web-Based Scalable Visualization of Large Document Collections](https://mlsanity.com/paper/2003.00975/) (2020)
- [ClioQuery: Interactive Query-Oriented Text Analytics for Comprehensive Investigation of Historical News Archives](https://mlsanity.com/paper/2204.04694/) (2022)
- [Supporting Serendipitous Discovery and Balanced Analysis of Online Product Reviews with Interaction-Driven Metrics and Bias-Mitigating Suggestions](https://mlsanity.com/paper/2202.06221/) (2022)

## Abstract (arXiv)

Appropriate evaluation and experimental design are fundamental for empirical sciences, particularly in data-driven fields. Due to the successes in computational modeling of languages, for instance, research outcomes are having an increasingly immediate impact on end users. As the gap in adoption by end users decreases, the need increases to ensure that tools and models developed by the research communities and practitioners are reliable, trustworthy, and supportive of the users in their goals. In this position paper, we focus on the issues of evaluating visual text analytics approaches. We take an interdisciplinary perspective from the visualization and natural language processing communities, as we argue that the design and validation of visual text analytics include concerns beyond computational or visual/interactive methods on their own. We identify four key groups of challenges for evaluating visual text analytics approaches (data ambiguity, experimental design, user trust, and "big picture" concerns) and provide suggestions for research opportunities from an interdisciplinary perspective.