# CRABS: A syntactic-semantic pincer strategy for bounding LLM interpretation of Python notebooks

arXiv: https://arxiv.org/abs/2507.11742 · HTML: https://mlsanity.com/paper/2507.11742/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 37 references that are in the map.
- First posted on arXiv 15 Jul 2025.

## Builds on (most cited first, 15 of 15)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020)
- [RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation](https://mlsanity.com/paper/2303.12570/) (2023)
- [Using an LLM to Help With Code Understanding](https://mlsanity.com/paper/2307.08177/) (2023)
- [Natural Language to Code Generation in Interactive Data Science Notebooks](https://mlsanity.com/paper/2212.09248/) (2022)
- [Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?](https://mlsanity.com/paper/2407.10956/) (2024)
- [Dataflow-Guided Retrieval Augmentation for Repository-Level Code Completion](https://mlsanity.com/paper/2405.19782/) (2024)
- [Execution-based Evaluation for Data Science Code Generation Models](https://mlsanity.com/paper/2211.09374/) (2022)
- [A large language model-assisted education tool to provide feedback on open-ended responses](https://mlsanity.com/paper/2308.02439/) (2023)
- [LLMDFA: Analyzing Dataflow in Code with Large Language Models](https://mlsanity.com/paper/2402.10754/) (2024)
- [Analyzing the Performance of Large Language Models on Code Summarization](https://mlsanity.com/paper/2404.08018/) (2024)
- [Python Code Generation by Asking Clarification Questions](https://mlsanity.com/paper/2212.09885/) (2022)
- [YesWorkflow: A User-Oriented, Language-Independent Tool for Recovering Workflow Information from Scripts](https://mlsanity.com/paper/1502.02403/) (2015)
- [Contextualized Data-Wrangling Code Generation in Computational Notebooks](https://mlsanity.com/paper/2409.13551/) (2024)
- [Suggesting Code Edits in Interactive Machine Learning Notebooks Using Large Language Models](https://mlsanity.com/paper/2501.09745/) (2025)

## Built on by (most cited first, 1 of 1)

- [PrepBench: How Far Are We from Natural-Language-Driven Data Preparation?](https://mlsanity.com/paper/2605.08687/) (2026)

## Abstract (arXiv)

Recognizing the information flows and operations comprising data science and machine learning Python notebooks is critical for evaluating, reusing, and adapting notebooks for new tasks. Investigating a notebook via re-execution often is impractical due to the challenges of resolving data and software dependencies. While Large Language Models (LLMs) pre-trained on large codebases have demonstrated effectiveness in understanding code without running it, we observe that they fail to understand some realistic notebooks due to hallucinations and long-context challenges. To address these issues, we propose a notebook understanding task yielding an information flow graph and corresponding cell execution dependency graph for a notebook, and demonstrate the effectiveness of a pincer strategy that uses limited syntactic analysis to assist full comprehension of the notebook using an LLM. Our Capture and Resolve Assisted Bounding Strategy (CRABS) employs shallow syntactic parsing and analysis of the abstract syntax tree (AST) to capture the correct interpretation of a notebook between lower and upper estimates of the inter-cell I/O setunicodex2014the flows of information into or out of cells via variablesunicodex2014then uses an LLM to resolve remaining ambiguities via cell-by-cell zero-shot learning, thereby identifying the true data inputs and outputs of each cell. We evaluate and demonstrate the effectiveness of our approach using an annotated dataset of 50 representative, highly up-voted Kaggle notebooks that together represent 3454 actual cell inputs and outputs. The LLM correctly resolves 1397 of 1425 (98%) ambiguities left by analyzing the syntactic structure of these notebooks. Across 50 notebooks, CRABS achieves average F1 scores of 98% identifying cell-to-cell information flows and 99% identifying transitive cell execution dependencies.