# Predicting the Understandability of Computational Notebooks through Code Metrics Analysis

arXiv: https://arxiv.org/abs/2406.10989 · HTML: https://mlsanity.com/paper/2406.10989/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 87 references that are in the map.
- First posted on arXiv 16 Jun 2024.

## Builds on (most cited first, 20 of 20)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [Unified Pre-training for Program Understanding and Generation](https://mlsanity.com/paper/2103.06333/) (2021)
- [Consistent Individualized Feature Attribution for Tree Ensembles](https://mlsanity.com/paper/1802.03888/) (2018)
- [On the Design of AI-powered Code Assistants for Notebooks](https://mlsanity.com/paper/2301.11178/) (2023)
- [Documentation Matters: Human-Centered AI System to Assist Data Science Code Documentation in Computational Notebooks](https://mlsanity.com/paper/2102.12592/) (2021)
- [Better Code, Better Sharing:On the Need of Analyzing Jupyter Notebooks](https://mlsanity.com/paper/1906.05234/) (2019)
- [KGTorrent: A Dataset of Python Jupyter Notebooks from Kaggle](https://mlsanity.com/paper/2103.10558/) (2021)
- [Evaluating Code Readability and Legibility: An Examination of Human-centric Studies](https://mlsanity.com/paper/2110.00785/) (2021)
- [A Large-Scale Comparison of Python Code in Jupyter Notebooks and Scripts](https://mlsanity.com/paper/2203.16718/) (2022)
- [HAConvGNN: Hierarchical Attention Based Convolutional Graph Neural Network for Code Documentation Generation in Jupyter Notebooks](https://mlsanity.com/paper/2104.01002/) (2021)
- [Restoring Execution Environments of Jupyter Notebooks](https://mlsanity.com/paper/2103.02959/) (2021)
- [Bug Analysis in Jupyter Notebook Projects: An Empirical Study](https://mlsanity.com/paper/2210.06893/) (2022)
- [ReSplit: Improving the Structure of Jupyter Notebooks by Re-Splitting Their Cells](https://mlsanity.com/paper/2112.14825/) (2021)
- [Behind the Scenes: On the Relationship Between Developer Experience and Refactoring](https://mlsanity.com/paper/2109.11089/) (2021)
- [Understanding the Characteristics of Visual Contents in Open Source Issue Discussions: A Case Study of Jupyter Notebook](https://mlsanity.com/paper/2204.13262/) (2022)
- [Assessing the Quality of Computational Notebooks for a Frictionless Transition from Exploration to Production](https://mlsanity.com/paper/2205.11941/) (2022)
- [Mining the Characteristics of Jupyter Notebooks in Data Science Projects](https://mlsanity.com/paper/2304.05325/) (2023)

## Abstract (arXiv)

Computational notebooks are the primary coding tools for data scientists, but their code quality remains understudied and often poor. Given the importance of maintainability and reusability, enhancing code understandability is essential. Traditional methods for assessing understandability typically rely on limited questionnaires or metadata like likes and votes, which may not reflect actual code clarity. To address this, we propose a novel approach that leverages user opinions from software repositories to assess the understandability of Jupyter notebooks. We conducted a case study using 542,051 Kaggle Jupyter notebooks compiled in the DistilKaggle dataset. To identify user comments related to code understandability, we used a fine-tuned DistilBERT transformer. We then introduced a new metric, i.e., User Opinion Code Understandability (UOCU), based on the number of relevant comments, their upvotes, and notebook views. UOCU proved significantly more effective than prior methods. We further enhanced it by combining UOCU with total upvotes in a hybrid approach. Using this improved metric, we collected 34 notebook-level metrics from 132,723 final notebooks and trained machine learning models to predict understandability. Our best model, a Random Forest classifier, achieved 89% accuracy in classifying the understandability level of notebook code. This work demonstrates the value of user opinion signals and notebook metrics in building scalable, accurate measures of code understandability.