# scicode-lint: Detecting Methodology Bugs in Scientific Python Code with LLM-Generated Patterns

arXiv: https://arxiv.org/abs/2603.17893 · HTML: https://mlsanity.com/paper/2603.17893/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 24 references that are in the map.
- First posted on arXiv 18 Mar 2026.

## Builds on (most cited first, 8 of 8)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Do Users Write More Insecure Code with AI Assistants?](https://mlsanity.com/paper/2211.03622/) (2022)
- [Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans](https://mlsanity.com/paper/2008.06388/) (2020)
- [KNighter: Transforming Static Analysis with LLM-Synthesized Checkers](https://mlsanity.com/paper/2503.09002/) (2025)
- [Data Leakage in Notebooks: Static Detection and Better Processes](https://mlsanity.com/paper/2209.03345/) (2022)
- [Abstract Interpretation-Based Data Leakage Static Analysis](https://mlsanity.com/paper/2211.16073/) (2022)
- [A survey of generative AI adoption and perceived productivity among scientists who program](https://mlsanity.com/paper/2512.19644/) (2025)
- [MLScent A tool for Anti-pattern detection in ML projects](https://mlsanity.com/paper/2502.18466/) (2025)

## Built on by (most cited first, 3 of 3)

- [Reward Hacking Challenges Oversight of Autonomous Research Agents](https://mlsanity.com/paper/2609.28614/) (2026)
- [sciwrite-lint: Verification Infrastructure for the Age of Science Vibe-Writing](https://mlsanity.com/paper/2604.08501/) (2026)
- [Rules to Tools: Executable Checks for LLM Agents in Scientific Computing](https://mlsanity.com/paper/2610.00313/) (2026)

## Abstract (arXiv)

Methodology bugs in scientific Python code produce plausible but incorrect results that traditional linters and static analysis tools cannot detect. Several research groups have built ML-specific linters, demonstrating that detection is feasible. Yet these tools share a sustainability problem: dependency on specific pylint or Python versions, limited packaging, and reliance on manual engineering for every new pattern. As AI-generated code increases the volume of scientific software, the need for automated methodology checking (such as detecting data leakage, incorrect cross-validation, and missing random seeds) grows. We present scicode-lint, whose two-tier architecture separates pattern design (frontier models at build time) from execution (small local model at runtime). Patterns are generated, not hand-coded; adapting to new library versions costs tokens, not engineering hours. On Kaggle notebooks with human-labeled ground truth, preprocessing leakage detection reaches 65% precision at 100% recall; on 38 published scientific papers applying AI/ML, precision is 62% (LLM-judged) with substantial variation across pattern categories; on a held-out paper set, precision is 54%. On controlled tests, scicode-lint achieves 97.7% accuracy across 66 patterns.