# OBsmith: LLM-Powered JavaScript Obfuscator Testing

arXiv: https://arxiv.org/abs/2510.10066 · HTML: https://mlsanity.com/paper/2510.10066/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 108 references that are in the map.
- First posted on arXiv 11 Oct 2025.

## Builds on (most cited first, 30 of 31)

- [StarCoder 2 and The Stack v2: The Next Generation](https://mlsanity.com/paper/2402.19173/) (2024)
- [Agentless: Demystifying LLM-based Software Engineering Agents](https://mlsanity.com/paper/2407.01489/) (2024)
- [Reinforcement learning](https://mlsanity.com/paper/2405.10369/) (2024)
- [Using an LLM to Help With Code Understanding](https://mlsanity.com/paper/2307.08177/) (2023)
- [CodePlan: Repository-level Coding using LLMs and Planning](https://mlsanity.com/paper/2309.12499/) (2023)
- [How well do Large Language Models perform in Arithmetic tasks?](https://mlsanity.com/paper/2304.02015/) (2023)
- [Baldur: Whole-Proof Generation and Repair with Large Language Models](https://mlsanity.com/paper/2303.04910/) (2023)
- [Fuzz4All: Universal Fuzzing with Large Language Models](https://mlsanity.com/paper/2308.04748/) (2023)
- [Code-Aware Prompting: A study of Coverage Guided Test Generation in Regression Setting using LLM](https://mlsanity.com/paper/2402.00097/) (2024)
- [TOGA: A Neural Method for Test Oracle Generation](https://mlsanity.com/paper/2109.09262/) (2021)
- [A Quantitative and Qualitative Evaluation of LLM-Based Explainable Fault Localization](https://mlsanity.com/paper/2308.05487/) (2023)
- [Make LLM a Testing Expert: Bringing Human-like Interaction to Mobile GUI Testing via Functionality-aware Decisions](https://mlsanity.com/paper/2310.15780/) (2023)
- [LILAC: Log Parsing using LLMs with Adaptive Parsing Cache](https://mlsanity.com/paper/2310.01796/) (2023)
- [LLMParser: An Exploratory Study on Using Large Language Models for Log Parsing](https://mlsanity.com/paper/2404.18001/) (2024)
- [Language Models Do Hard Arithmetic Tasks Easily and Hardly Do Easy Arithmetic Tasks](https://mlsanity.com/paper/2406.02356/) (2024)
- [Unprecedented Code Change Automation: The Fusion of LLMs and Transformation by Example](https://mlsanity.com/paper/2402.07138/) (2024)
- [LLM-Powered Test Case Generation for Detecting Bugs in Plausible Programs](https://mlsanity.com/paper/2404.10304/) (2024)
- [Rocks Coding, Not Development--A Human-Centric, Experimental Evaluation of LLM-Supported SE Tasks](https://mlsanity.com/paper/2402.05650/) (2024)
- [Superion: Grammar-Aware Greybox Fuzzing](https://mlsanity.com/paper/1812.01197/) (2018)
- [Mate! Are You Really Aware? An Explainability-Guided Testing Framework for Robustness of Malware Detectors](https://mlsanity.com/paper/2111.10085/) (2021)
- [On the Effectiveness of Large Language Models in Writing Alloy Formulas](https://mlsanity.com/paper/2502.15441/) (2025)
- [Skeletal Program Enumeration for Rigorous Compiler Testing](https://mlsanity.com/paper/1610.03148/) (2016)
- [Generating executable oracles to check conformance of client code to requirements of JDK Javadocs using LLMs](https://mlsanity.com/paper/2411.01789/) (2024)
- [UNIFUZZ: A Holistic and Pragmatic Metrics-Driven Platform for Evaluating Fuzzers](https://mlsanity.com/paper/2010.01785/) (2020)
- [Uncovering the Causes of Emotions in Software Developer Communication Using Zero-shot LLMs](https://mlsanity.com/paper/2312.09731/) (2023)
- [CASCADE: LLM-Powered JavaScript Deobfuscator at Google](https://mlsanity.com/paper/2507.17691/) (2025)
- [JSKETCH: Sketching for Java](https://mlsanity.com/paper/1507.03577/) (2015)
- [FV8: A Forced Execution JavaScript Engine for Detecting Evasive Techniques](https://mlsanity.com/paper/2405.13175/) (2024)
- [Interleaving Large Language Models for Compiler Testing](https://mlsanity.com/paper/2508.18955/) (2025)
- [Java JIT Testing with Template Extraction](https://mlsanity.com/paper/2403.11281/) (2024)

## Built on by (most cited first, 2 of 2)

- [Semantic Voting: Execution-Grounded Consensus for LLM Code Generation](https://mlsanity.com/paper/2605.08680/) (2026)
- [Sketch-and-Verify: Structured Inference-Time Scaling via Program Sketching](https://mlsanity.com/paper/2605.08658/) (2026)

## Abstract (arXiv)

JavaScript obfuscators are widely deployed to protect intellectual property and resist reverse engineering, yet their correctness has been largely overlooked compared to performance and resilience. Existing evaluations typically measure resistance to deobfuscation, leaving the critical question of whether obfuscators preserve program semantics unanswered. Incorrect transformations can silently alter functionality, compromise reliability, and erode security-undermining the very purpose of obfuscation. To address this gap, we present OBsmith, a novel framework to systematically test JavaScript obfuscators using large language models (LLMs). OBsmith leverages LLMs to generate program sketches abstract templates capturing diverse language constructs, idioms, and corner cases-which are instantiated into executable programs and subjected to obfuscation under different configurations. Besides LLM-powered sketching, OBsmith also employs a second source: automatic extraction of sketches from real programs. This extraction path enables more focused testing of project specific features and lets developers inject domain knowledge into the resulting test cases. OBsmith uncovers 11 previously unknown correctness bugs. Under an equal program budget, five general purpose state-of-the-art JavaScript fuzzers (FuzzJIT, Jsfunfuzz, Superion, DIE, Fuzzilli) failed to detect these issues, highlighting OBsmith's complementary focus on obfuscation induced misbehavior. An ablation shows that all components except our generic MRs contribute to at least one bug class; the negative MR result suggests the need for obfuscator-specific metamorphic relations. Our results also seed discussion on how to balance obfuscation presets and performance cost. We envision OBsmith as an important step towards automated testing and quality assurance of obfuscators and other semantic-preserving toolchains.