# Generating Java Methods: An Empirical Assessment of Four AI-Based Code Assistants

arXiv: https://arxiv.org/abs/2402.08431 · HTML: https://mlsanity.com/paper/2402.08431/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Feb 2024.

## Built on by (most cited first, 6 of 6)

- [Agents4PLC: Automating Closed-loop PLC Code Generation and Verification in Industrial Control Systems using LLM-based Agents](https://mlsanity.com/paper/2410.14209/) (2024)
- [The Impact of Prompt Programming on Function-Level Code Generation](https://mlsanity.com/paper/2412.20545/) (2024)
- [AI IDEs or Autonomous Agents? Measuring the Impact of Coding Agents on Software Development](https://mlsanity.com/paper/2601.13597/) (2026)
- [Benchmarking LLM Code Generation for Audio Programming with Visual Dataflow Languages](https://mlsanity.com/paper/2409.00856/) (2024)
- [From Overload to Insights: How AI Agents Can Support Scientists in Analyzing Complex Data](https://mlsanity.com/paper/2607.16845/) (2026)
- [Generation of Web Apps with Agentic IDEs: An Empirical Assessment](https://mlsanity.com/paper/2608.20903/) (2026)

## Abstract (arXiv)

AI-based code assistants are promising tools that can facilitate and speed up code development. They exploit machine learning algorithms and natural language processing to interact with developers, suggesting code snippets (e.g., method implementations) that can be incorporated into projects. Recent studies empirically investigated the effectiveness of code assistants using simple exemplary problems (e.g., the re-implementation of well-known algorithms), which fail to capture the spectrum and nature of the tasks actually faced by developers. In this paper, we expand the knowledge in the area by comparatively assessing four popular AI-based code assistants, namely GitHub Copilot, Tabnine, ChatGPT, and Google Bard, with a dataset of 100 methods that we constructed from real-life open-source Java projects, considering a variety of cases for complexity and dependency from contextual elements. Results show that Copilot is often more accurate than other techniques, yet none of the assistants is completely subsumed by the rest of the approaches. Interestingly, the effectiveness of these solutions dramatically decreases when dealing with dependencies outside the boundaries of single classes.