# CodeWatcher: IDE Telemetry Data Extraction Tool for Understanding Coding Interactions with LLMs

arXiv: https://arxiv.org/abs/2510.11536 · HTML: https://mlsanity.com/paper/2510.11536/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 22 references that are in the map.
- First posted on arXiv 13 Oct 2025.

## Builds on (most cited first, 10 of 10)

- [Grounded Copilot: How Programmers Interact with Code-Generating Models](https://mlsanity.com/paper/2206.15000/) (2022)
- [In-IDE Code Generation from Natural Language: Promise and Challenges](https://mlsanity.com/paper/2101.11149/) (2021)
- [WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code Visualization](https://mlsanity.com/paper/2408.01703/) (2024)
- [Investigating and Designing for Trust in AI-powered Code Generation Tools](https://mlsanity.com/paper/2305.11248/) (2023)
- [What Makes a Code Review Useful to OpenDev Developers? An Empirical Investigation](https://mlsanity.com/paper/2302.11686/) (2023)
- [Is This You, LLM? Recognizing AI-written Programs with Multilingual Code Stylometry](https://mlsanity.com/paper/2412.14611/) (2024)
- [A Case Study of Onboarding in Software Teams: Tasks and Strategies](https://mlsanity.com/paper/2103.05055/) (2021)
- [TaskTracker-tool: a Toolkit for Tracking of Code Snapshots and Activity Data During Solution of Programming Tasks](https://mlsanity.com/paper/2012.05085/) (2020)
- [What we can learn from how programmers debug their code](https://mlsanity.com/paper/2103.12447/) (2021)
- [What's DAT? Three Case Studies of Measuring Software Development Productivity at Meta With Diff Authoring Time](https://mlsanity.com/paper/2503.10977/) (2025)

## Built on by (most cited first, 2 of 2)

- [RECAP: An End-to-End Platform for Capturing, Replaying, and Analyzing AI-Assisted Programming Interactions](https://mlsanity.com/paper/2605.01104/) (2026)
- [Comprendia: AI-Augmented Code Comprehension](https://mlsanity.com/paper/2608.10290/) (2026)

## Abstract (arXiv)

Understanding how developers interact with code generation tools (CGTs) requires detailed, real-time data on programming behavior which is often difficult to collect without disrupting workflow. We present textitCodeWatcher, a lightweight, unobtrusive client-server system designed to capture fine-grained interaction events from within the Visual Studio Code (VS Code) editor. textitCodeWatcher logs semantically meaningful events such as insertions made by CGTs, deletions, copy-paste actions, and focus shifts, enabling continuous monitoring of developer activity without modifying user workflows. The system comprises a VS Code plugin, a Python-based RESTful API, and a MongoDB backend, all containerized for scalability and ease of deployment. By structuring and timestamping each event, textitCodeWatcher enables post-hoc reconstruction of coding sessions and facilitates rich behavioral analyses, including how and when CGTs are used during development. This infrastructure is crucial for supporting research on responsible AI, developer productivity, and the human-centered evaluation of CGTs. Please find the demo, diagrams, and tool here: https://osf.io/j2kru/overview.