# Modeling Disclosive Transparency in NLP Application Descriptions

arXiv: https://arxiv.org/abs/2101.00433 · HTML: https://mlsanity.com/paper/2101.00433/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 43 references that are in the map.
- First posted on arXiv 2 Jan 2021.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Manipulating and Measuring Model Interpretability](https://mlsanity.com/paper/1802.07810/) (2018)
- [Explainable Machine Learning in Deployment](https://mlsanity.com/paper/1909.06342/) (2019)
- [Troubling Trends in Machine Learning Scholarship](https://mlsanity.com/paper/1807.03341/) (2018)
- [The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards](https://mlsanity.com/paper/1805.03677/) (2018)
- [Towards the Science of Security and Privacy in Machine Learning](https://mlsanity.com/paper/1611.03814/) (2016)
- [Learning and Evaluating General Linguistic Intelligence](https://mlsanity.com/paper/1901.11373/) (2019)
- [Human Factors in Model Interpretability: Industry Practices, Challenges, and Needs](https://mlsanity.com/paper/2004.11440/) (2020)
- [Differential Privacy and Machine Learning: a Survey and Review](https://mlsanity.com/paper/1412.7584/) (2014)
- [Fairness and Accountability Design Needs for Algorithmic Support in High-Stakes Public Sector Decision-Making](https://mlsanity.com/paper/1802.01029/) (2018)
- [Factors Influencing Perceived Fairness in Algorithmic Decision-Making: Algorithm Outcomes, Development Procedures, and Individual Differences](https://mlsanity.com/paper/2001.09604/) (2020)
- [Semantic Complexity in End-to-End Spoken Language Understanding](https://mlsanity.com/paper/2008.02858/) (2020)
- [Robust Estimation of Hypernasality in Dysarthria with Acoustic Model Likelihood Features](https://mlsanity.com/paper/1911.11360/) (2019)

## Built on by (most cited first, 4 of 4)

- [Not All Errors are Equal: Learning Text Generation Metrics using Stratified Error Synthesis](https://mlsanity.com/paper/2210.05035/) (2022)
- [ChatGPT Rates Natural Language Explanation Quality Like Humans: But on Which Scales?](https://mlsanity.com/paper/2403.17368/) (2024)
- [Users are the North Star for AI Transparency](https://mlsanity.com/paper/2303.05500/) (2023)
- [Gradient-Based Word Substitution for Obstinate Adversarial Examples Generation in Language Models](https://mlsanity.com/paper/2307.12507/) (2023)

## Abstract (arXiv)

Broader disclosive transparency-truth and clarity in communication regarding the function of AI systems-is widely considered desirable. Unfortunately, it is a nebulous concept, difficult to both define and quantify. This is problematic, as previous work has demonstrated possible trade-offs and negative consequences to disclosive transparency, such as a confusion effect, where "too much information" clouds a reader's understanding of what a system description means. Disclosive transparency's subjective nature has rendered deep study into these problems and their remedies difficult. To improve this state of affairs, We introduce neural language model-based probabilistic metrics to directly model disclosive transparency, and demonstrate that they correlate with user and expert opinions of system transparency, making them a valid objective proxy. Finally, we demonstrate the use of these metrics in a pilot study quantifying the relationships between transparency, confusion, and user perceptions in a corpus of real NLP system descriptions.