# WatChat: Explaining perplexing programs by debugging mental models

arXiv: https://arxiv.org/abs/2403.05334 · HTML: https://mlsanity.com/paper/2403.05334/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 108 references that are in the map.
- First posted on arXiv 8 Mar 2024.

## Builds on (most cited first, 8 of 8)

- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Evaluating Language Models for Mathematics through Interactions](https://mlsanity.com/paper/2306.01694/) (2023)
- [Selective Explanations: Leveraging Human Input to Align Explainable AI](https://mlsanity.com/paper/2301.09656/) (2023)
- [Inferring the Future by Imagining the Past](https://mlsanity.com/paper/2305.17195/) (2023)
- [A Grounded Conceptual Model for Ownership Types in Rust](https://mlsanity.com/paper/2309.04134/) (2023)
- [The Essence of JavaScript](https://mlsanity.com/paper/1510.00925/) (2015)
- [Bonsai: Synthesis-Based Reasoning for Type Systems](https://mlsanity.com/paper/1708.00551/) (2017)
- [It's Like Python But: Towards Supporting Transfer of Programming Language Knowledge](https://mlsanity.com/paper/1808.09008/) (2018)

## Built on by (most cited first, 3 of 3)

- [Building Machines that Learn and Think with People](https://mlsanity.com/paper/2408.03943/) (2024)
- [Retrieval-augmented systems can be dangerous medical communicators](https://mlsanity.com/paper/2502.14898/) (2025)
- [Empathy in Explanation](https://mlsanity.com/paper/2507.21081/) (2025)

## Abstract (arXiv)

Often, a good explanation for a program's unexpected behavior is a bug in the programmer's code. But sometimes, an even better explanation is a bug in the programmer's mental model of the language or API they are using. Instead of merely debugging our current code ("giving the programmer a fish"), what if our tools could directly debug our mental models ("teaching the programmer to fish")? In this paper, we apply recent ideas from computational cognitive science to offer a principled framework for doing exactly that. Given a "why?" question about a program, we automatically infer potential misconceptions about the language/API that might cause the user to be surprised by the program's behavior -- and then analyze those misconceptions to provide explanations of the program's behavior. Our key idea is to formally represent misconceptions as counterfactual (erroneous) semantics for the language/API, which can be inferred and debugged using program synthesis techniques. We demonstrate our framework, WatChat, by building systems for explanation in two domains: JavaScript type coercion, and the Git version control system. We evaluate WatChatJS and WatChatGit by comparing their outputs to experimentally-collected human-written explanations in these two domains: we show that WatChat's explanations exhibit key features of human-written explanation, unlike those of a state-of-the-art language model.