# Illuminating LLM Coding Agents: Visual Analytics for Deeper Understanding and Enhancement

arXiv: https://arxiv.org/abs/2508.12555 · HTML: https://mlsanity.com/paper/2508.12555/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 37 of its 75 references that are in the map.
- First posted on arXiv 18 Aug 2025.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 30 of 37)

- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering](https://mlsanity.com/paper/2410.07095/) (2024)
- [CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning](https://mlsanity.com/paper/2207.01780/) (2022)
- [CodeT5+: Open Code Large Language Models for Code Understanding and Generation](https://mlsanity.com/paper/2305.07922/) (2023)
- [Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training](https://mlsanity.com/paper/2309.17179/) (2023)
- [AIDE: AI-Driven Exploration in the Space of Code](https://mlsanity.com/paper/2502.13138/) (2025)
- [The What-If Tool: Interactive Probing of Machine Learning Models](https://mlsanity.com/paper/1907.04135/) (2019)
- [RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts](https://mlsanity.com/paper/2411.15114/) (2024)
- [Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers](https://mlsanity.com/paper/1801.06889/) (2018)
- [ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing](https://mlsanity.com/paper/2309.09128/) (2023)
- [EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteria](https://mlsanity.com/paper/2309.13633/) (2023)
- [AI Research Agents for Machine Learning: Search, Exploration, and Generalization in MLE-bench](https://mlsanity.com/paper/2507.02554/) (2025)
- [Seq2Seq-Vis: A Visual Debugging Tool for Sequence-to-Sequence Models](https://mlsanity.com/paper/1804.09299/) (2018)
- [Do Convolutional Neural Networks Learn Class Hierarchy?](https://mlsanity.com/paper/1710.06501/) (2017)
- [ML-Master: Towards AI-for-AI via Integration of Exploration and Reasoning](https://mlsanity.com/paper/2506.16499/) (2025)
- [CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modules](https://mlsanity.com/paper/2310.08992/) (2023)
- [Towards Better Analysis of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1604.07043/) (2016)
- [Wider or Deeper? Scaling LLM Inference-Time Compute with Adaptive Branching Tree Search](https://mlsanity.com/paper/2503.04412/) (2025)
- [InSituNet: Deep Image Synthesis for Parameter Space Exploration of Ensemble Simulations](https://mlsanity.com/paper/1908.00407/) (2019)
- [A Generative Model for Volume Rendering](https://mlsanity.com/paper/1710.09545/) (2017)
- [How Does Attention Work in Vision Transformers? A Visual Analytics Attempt](https://mlsanity.com/paper/2303.13731/) (2023)
- [LLM Comparator: Visual Analytics for Side-by-Side Evaluation of Large Language Models](https://mlsanity.com/paper/2402.10524/) (2024)
- [A Survey on ML4VIS: Applying Machine Learning Advances to Data Visualization](https://mlsanity.com/paper/2012.00467/) (2020)
- [AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunities](https://mlsanity.com/paper/2508.11126/) (2025)
- [Visual Analytics For Machine Learning: A Data Perspective Survey](https://mlsanity.com/paper/2307.07712/) (2023)
- [A Deep Generative Model for Graph Layout](https://mlsanity.com/paper/1904.12225/) (2019)
- [DeepDrawing: A Deep Learning Approach to Graph Drawing](https://mlsanity.com/paper/1907.11040/) (2019)

## Built on by (most cited first, 3 of 3)

- [What Resolve Rate Hides: Trajectory Structure Diagnostics for Coding Agents](https://mlsanity.com/paper/2607.06184/) (2026)
- [TraceView: Interactive Visualization of Agentic Program Repair Trajectories](https://mlsanity.com/paper/2606.22110/) (2026)
- [Token Signatures of Code: Comparing Coding Behaviors Across Large Language Models](https://mlsanity.com/paper/2609.22097/) (2026)

## Abstract (arXiv)

Coding agents powered by large language models (LLMs) have gained traction for automating code generation through iterative problem-solving with minimal human involvement. Despite the emergence of various frameworks, e.g., LangChain, AutoML, and AIDE, ML scientists still struggle to effectively review and adjust the agents' coding process. The current approach of manually inspecting individual outputs is inefficient, making it difficult to track code evolution, compare coding iterations, and identify improvement opportunities. To address this challenge, we introduce a visual analytics system designed to enhance the examination of coding agent behaviors. Focusing on the AIDE framework, our system supports comparative analysis across three levels: (1) Code-Level Analysis, which reveals how the agent debugs and refines its code over iterations; (2) Process-Level Analysis, which contrasts different solution-seeking processes explored by the agent; and (3) LLM-Level Analysis, which highlights variations in coding behavior across different LLMs. By integrating these perspectives, our system enables ML scientists to gain a structured understanding of agent behaviors, facilitating more effective debugging and prompt engineering. Through case studies using coding agents to tackle popular Kaggle competitions, we demonstrate how our system provides valuable insights into the iterative coding process.