# Opening up ChatGPT: Tracking openness, transparency, and accountability in instruction-tuned text generators

arXiv: https://arxiv.org/abs/2307.05532 · HTML: https://mlsanity.com/paper/2307.05532/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 27 of its 65 references that are in the map.
- First posted on arXiv 8 Jul 2023.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 27 of 27)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Augmented Language Models: a Survey](https://mlsanity.com/paper/2302.07842/) (2023)
- [A Systematic Evaluation of Large Language Models of Code](https://mlsanity.com/paper/2202.13169/) (2022)
- [Multimodal datasets: misogyny, pornography, and malignant stereotypes](https://mlsanity.com/paper/2110.01963/) (2021)
- [Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model](https://mlsanity.com/paper/2211.02001/) (2022)
- [Data and its (dis)contents: A survey of dataset development and use in machine learning research](https://mlsanity.com/paper/2012.05345/) (2020)
- [Large image datasets: A pyrrhic win for computer vision?](https://mlsanity.com/paper/2006.16923/) (2020)
- [Trustworthy AI: From Principles to Practices](https://mlsanity.com/paper/2110.01167/) (2021)
- [The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset](https://mlsanity.com/paper/2303.03915/) (2023)
- [Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure](https://mlsanity.com/paper/2010.13561/) (2020)
- [Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces](https://mlsanity.com/paper/1709.10163/) (2017)
- [Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset](https://mlsanity.com/paper/2207.00220/) (2022)
- [The Gradient of Generative AI Release: Methods and Considerations](https://mlsanity.com/paper/2302.04844/) (2023)
- [Interactive Model Cards: A Human-Centered Approach to Model Documentation](https://mlsanity.com/paper/2205.02894/) (2022)
- [Collaboration Challenges in Building ML-Enabled Systems: Communication, Documentation, Engineering, and Process](https://mlsanity.com/paper/2110.10234/) (2021)
- [Changing the World by Changing the Data](https://mlsanity.com/paper/2105.13947/) (2021)
- [Reusable Templates and Guides For Documenting Datasets and Models for Natural Language Processing and Generation: A Case Study of the HuggingFace and GEM Data and Model Cards](https://mlsanity.com/paper/2108.07374/) (2021)
- [Towards decolonising computational sciences](https://mlsanity.com/paper/2009.14258/) (2020)
- [EleutherAI: Going Beyond "Open Science" to "Science in the Open"](https://mlsanity.com/paper/2210.06413/) (2022)
- [Dynamic Planning in Open-Ended Dialogue using Reinforcement Learning](https://mlsanity.com/paper/2208.02294/) (2022)

## Built on by (most cited first, 30 of 56)

- [Unintended Impacts of LLM Alignment on Global Representation](https://mlsanity.com/paper/2402.15018/) (2024)
- [How Linguistics Learned to Stop Worrying and Love the Language Models](https://mlsanity.com/paper/2501.17047/) (2025)
- [Trust No Bot: Discovering Personal Disclosures in Human-LLM Conversations in the Wild](https://mlsanity.com/paper/2407.11438/) (2024)
- [Best Practices for Text Annotation with Large Language Models](https://mlsanity.com/paper/2402.05129/) (2024)
- [Reproducibility in Machine Learning-based Research: Overview, Barriers and Drivers](https://mlsanity.com/paper/2406.14325/) (2024)
- [Navigating LLM Ethics: Advancements, Challenges, and Future Directions](https://mlsanity.com/paper/2406.18841/) (2024)
- [Empirical evidence of Large Language Model's influence on human spoken communication](https://mlsanity.com/paper/2409.01754/) (2024)
- [The Moral Machine Experiment on Large Language Models](https://mlsanity.com/paper/2309.05958/) (2023)
- [The Model Openness Framework: Promoting Completeness and Openness for Reproducibility, Transparency, and Usability in Artificial Intelligence](https://mlsanity.com/paper/2403.13784/) (2024)
- [The AI Community Building the Future? A Quantitative Analysis of Development Activity on Hugging Face Hub](https://mlsanity.com/paper/2405.13058/) (2024)
- [Large Action Models: From Inception to Implementation](https://mlsanity.com/paper/2412.10047/) (2024)
- [The Open Source Advantage in Large Language Models (LLMs)](https://mlsanity.com/paper/2412.12004/) (2024)
- [A Sentence is Worth a Thousand Pictures: Can Large Language Models Understand Hum4n L4ngu4ge and the W0rld behind W0rds?](https://mlsanity.com/paper/2308.00109/) (2023)
- [Machine-assisted quantitizing designs: augmenting humanities and social sciences with artificial intelligence](https://mlsanity.com/paper/2309.14379/) (2023)
- [Multi-party Goal Tracking with LLMs: Comparing Pre-training, Fine-tuning, and Prompt Engineering](https://mlsanity.com/paper/2308.15231/) (2023)
- [Near to Mid-term Risks and Opportunities of Open-Source Generative AI](https://mlsanity.com/paper/2404.17047/) (2024)
- [Risks and Opportunities of Open-Source Generative AI](https://mlsanity.com/paper/2405.08597/) (2024)
- [Language Models as a Service: Overview of a New Paradigm and its Challenges](https://mlsanity.com/paper/2309.16573/) (2023)
- [Why Linguistics Will Thrive in the 21st Century: A Reply to Piantadosi (2023)](https://mlsanity.com/paper/2308.03228/) (2023)
- [Laboratory-Scale AI: Open-Weight Models are Competitive with ChatGPT Even in Low-Resource Settings](https://mlsanity.com/paper/2405.16820/) (2024)
- [Aligning AI with Public Values: Deliberation and Decision-Making for Governing Multimodal LLMs in Political Video Analysis](https://mlsanity.com/paper/2410.01817/) (2024)
- [MusGO: A Community-Driven Framework For Assessing Openness in Music-Generative AI](https://mlsanity.com/paper/2507.03599/) (2025)
- [On the Workflows and Smells of Leaderboard Operations (LBOps): An Exploratory Study of Foundation Model Leaderboards](https://mlsanity.com/paper/2407.04065/) (2024)
- [Awes, Laws, and Flaws From Today's LLM Research](https://mlsanity.com/paper/2408.15409/) (2024)
- [To Err is AI : A Case Study Informing LLM Flaw Reporting Practices](https://mlsanity.com/paper/2410.12104/) (2024)
- [Had enough of experts? Quantitative knowledge retrieval from large language models](https://mlsanity.com/paper/2402.07770/) (2024)
- [Why Companies "Democratise" Artificial Intelligence: The Case of Open Source Software Donations](https://mlsanity.com/paper/2409.17876/) (2024)
- [ChapGTP, ILLC's Attempt at Raising a BabyLM: Improving Data Efficiency by Automatic Task Formation](https://mlsanity.com/paper/2310.11282/) (2023)
- [Pretrained Generative Language Models as General Learning Frameworks for Sequence-Based Tasks](https://mlsanity.com/paper/2402.05616/) (2024)
- [An evaluation of LLMs and Google Translate for translation of selected Indian languages via sentiment and semantic analyses](https://mlsanity.com/paper/2503.21393/) (2025)

## Abstract (arXiv)

Large language models that exhibit instruction-following behaviour represent one of the biggest recent upheavals in conversational interfaces, a trend in large part fuelled by the release of OpenAI's ChatGPT, a proprietary large language model for text generation fine-tuned through reinforcement learning from human feedback (LLM+RLHF). We review the risks of relying on proprietary software and survey the first crop of open-source projects of comparable architecture and functionality. The main contribution of this paper is to show that openness is differentiated, and to offer scientific documentation of degrees of openness in this fast-moving field. We evaluate projects in terms of openness of code, training data, model weights, RLHF data, licensing, scientific documentation, and access methods. We find that while there is a fast-growing list of projects billing themselves as 'open source', many inherit undocumented data of dubious legality, few share the all-important instruction-tuning (a key site where human annotation labour is involved), and careful scientific documentation is exceedingly rare. Degrees of openness are relevant to fairness and accountability at all points, from data collection and curation to model architecture, and from training and fine-tuning to release and deployment.