# Automated Conformance Testing for JavaScript Engines via Deep Compiler Fuzzing

arXiv: https://arxiv.org/abs/2104.07460 · HTML: https://mlsanity.com/paper/2104.07460/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Apr 2021.

## Built on by (most cited first, 12 of 12)

- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [Large Language Models for Compiler Optimization](https://mlsanity.com/paper/2309.07062/) (2023)
- [NNSmith: Generating Diverse and Valid Test Cases for Deep Learning Compilers](https://mlsanity.com/paper/2207.13066/) (2022)
- [CodeFuse-13B: A Pretrained Multi-lingual Code Large Language Model](https://mlsanity.com/paper/2310.06266/) (2023)
- [Compiler generated feedback for Large Language Models](https://mlsanity.com/paper/2403.14714/) (2024)
- [CovRL: Fuzzing JavaScript Engines with Coverage-Guided Reinforcement Learning for LLM-based Mutation](https://mlsanity.com/paper/2402.12222/) (2024)
- [Beyond Problem Solving: UOJ-Bench for Evaluating Code Generation, Hacking, and Repair in Competitive Programming](https://mlsanity.com/paper/2606.12864/) (2026)
- [An Empirical Study of Bugs in Data Visualization Libraries](https://mlsanity.com/paper/2506.15084/) (2025)
- [Agentic Fuzzing: Opportunities and Challenges](https://mlsanity.com/paper/2605.10074/) (2026)
- [Once4All: Skeleton-Guided SMT Solver Fuzzing with LLM-Synthesized Generators](https://mlsanity.com/paper/2508.20340/) (2025)
- [From Coverage to Causes: Data-Centric Fuzzing for JavaScript Engines](https://mlsanity.com/paper/2512.18102/) (2025)
- [SmartOracle -- An Agentic Approach to Mitigate Noise in Differential Oracles](https://mlsanity.com/paper/2601.15074/) (2026)

## Abstract (arXiv)

JavaScript (JS) is a popular, platform-independent programming language. To ensure the interoperability of JS programs across different platforms, the implementation of a JS engine should conform to the ECMAScript standard. However, doing so is challenging as there are many subtle definitions of API behaviors, and the definitions keep evolving. We present COMFORT, a new compiler fuzzing framework for detecting JS engine bugs and behaviors that deviate from the ECMAScript standard. COMFORT leverages the recent advance in deep learning-based language models to automatically generate JS test code. As a departure from prior fuzzers, COMFORT utilizes the well-structured ECMAScript specifications to automatically generate test data along with the test programs to expose bugs that could be overlooked by the developers or manually written test cases. COMFORT then applies differential testing methodologies on the generated test cases to expose standard conformance bugs. We apply COMFORT to ten mainstream JS engines. In 200 hours of automated concurrent testing runs, we discover bugs in all tested JS engines. We had identified 158 unique JS engine bugs, of which 129 have been verified, and 115 have already been fixed by the developers. Furthermore, 21 of the Comfort-generated test cases have been added to Test262, the official ECMAScript conformance test suite.