mlsanityData to 14 Oct 2026

VibeJam: A User Study Platform for Web Development with Agents

arXiv 2608.29889 ↗First posted 30 Aug 2026cs.CLcited by 0

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Programming with AI is increasingly agentic, users prompt LLMs to directly edit their code and review the changes, with adoption growing especially for web development tasks. Despite this growth, most NLP work uses offline evaluation and lacks support for online studies, losing insights into how programmers truly use coding agents. We release VibeJam, a browser-based user study platform for users to collaborate with AI agents to develop websites. VibeJam enables agent customization and uses the open-source Aider agent by default, and to mirror downstream use, we add diff review, chat and plan modes, and live website previews. In a pilot study with 55 released, game-based website creation tasks, five experienced AI programmers rate our system as fun, simple, and resembling commercial tools, while 13 junior students use VibeJam to make websites of higher quality than agents in the same task. We open-source VibeJam to spur extensions and support studies on how coding agents can help users.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 33 of its 63 references that are in the map. First posted on arXiv 30 Aug 2026.

Builds on

33 of 63 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Dive into Claude Code: The Design Space of Today's and Future AI Agent Systems14 Apr
  2. 2026GameDevBench: Evaluating Agentic Capabilities Through Game Development11 Feb
  3. 2025Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflows10 Jul

Other references · 30

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  4. 2022Grounded Copilot: How Programmers Interact with Code-Generating Modelscited by 218
  5. 2024Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineeringcited by 152
  6. 2023BooookScore: A systematic exploration of book-length summarization in the era of LLMscited by 137
  7. 2023CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classescited by 72
  8. 2025The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMscited by 64
  9. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 52
  10. 2022Interactive Code Generation via Test-Driven User-Intent Formalizationcited by 50
  11. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  12. 2024Need Help? Designing Proactive AI Assistants for Programmingcited by 43
18 more
  1. 2025Assistance or Disruption? Exploring and Evaluating the Design and Trade-offs of Proactive AI Programming Supportcited by 34
  2. 2024The RealHumanEval: Evaluating Large Language Models' Abilities to Support Programmerscited by 30
  3. 2025Copilot Arena: A Platform for Code LLM Evaluation in the Wildcited by 26
  4. 2025AI-Slop to AI-Polish? Aligning Language Models through Edit-Based Writing Rewards and Test-time Computationcited by 25
  5. 2024What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Usecited by 24
  6. 2025FrontendBench: A Benchmark for Evaluating LLMs on Front-End Development via Automatic Evaluationcited by 20
  7. 2024CS1-LLM: Integrating LLMs into CS1 Instructioncited by 16
  8. 2024Exploring the Design Space of Cognitive Engagement Techniques with AI-Generated Code for Enhanced Learningcited by 16
  9. 2024Ivie: Lightweight Anchored Explanations of Just-Generated Codecited by 14
  10. 2025CodingGenie: A Proactive LLM-Powered Programming Assistantcited by 10
  11. 2025Assessing the Quality and Security of AI-Generated Code: A Quantitative Analysiscited by 7
  12. 2025A Good Plan is Hard to Find: Aligning Models with Preferences is Misaligned with What Helps Userscited by 6
  13. 2025Building Software by Rolling the Dice: A Qualitative Study of Vibe Codingcited by 5
  14. 2026BenchMarker: An Education-Inspired Toolkit for Highlighting Flaws in Multiple-Choice Benchmarkscited by 5
  15. 2025Test-Time Reasoners Are Strategic Multiple-Choice Test-Takerscited by 3
  16. 2025Assessing Problem Decomposition in CS1 for the GenAI Eracited by 3
  17. 2026Editrail: Understanding AI Usage by Visualizing Student-AI Interaction in Codecited by 3
  18. 2026Why Human Guidance Matters in Collaborative Vibe Codingcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025AI Agents for Web Testing: A Case Study in the Wild
  2. 2025Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI
  3. 2026A Visionary Look at Vibe Researching
  4. 2025Vibe Coding: Toward an AI-Native Paradigm for Semantic and Intent-Driven Programming ↗
  5. 2026SWE-WebDevBench: Evaluating Coding Agent Application Platforms as Virtual Software Agencies
  6. 2025A Survey of Vibe Coding with Large Language Models
  7. 2025AgentA/B: Automated and Scalable Web A/BTesting with Interactive LLM Agents
  8. 2024The BrowserGym Ecosystem for Web Agent Research