mlsanityData to 14 Oct 2026

Evaluating Large Language Models in Code Generation: INFINITE Methodology for Defining the Inference Index

arXiv 2503.05852 ↗First posted 7 Mar 2025cs.SE cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

This study introduces a new methodology for an Inference Index (InI), called INFerence INdex In Testing model Effectiveness methodology (INFINITE), aiming to evaluate the performance of Large Language Models (LLMs) in code generation tasks. The InI index provides a comprehensive assessment focusing on three key components: efficiency, consistency, and accuracy. This approach encapsulates time-based efficiency, response quality, and the stability of model outputs, offering a thorough understanding of LLM performance beyond traditional accuracy metrics. We applied this methodology to compare OpenAI's GPT-4o (GPT), OpenAI-o1 pro (OAI1), and OpenAI-o3 mini-high (OAI3) in generating Python code for the Long-Short-Term-Memory (LSTM) model to forecast meteorological variables such as temperature, relative humidity and wind velocity. Our findings demonstrate that GPT outperforms OAI1 and performs comparably to OAI3 regarding accuracy and workflow efficiency. The study reveals that LLM-assisted code generation can produce results similar to expert-designed models with effective prompting and refinement. GPT's performance advantage highlights the benefits of widespread use and user feedback.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 62 references that are in the map. First posted on arXiv 7 Mar 2025.

Builds on

34 of 62 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  6. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  7. 2023Time-LLM: Time Series Forecasting by Reprogramming Large Language Modelscited by 700
  8. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 688
  9. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  10. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 479
  11. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 384
  12. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 334
22 more
  1. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 232
  2. 2023Using an LLM to Help With Code Understandingcited by 228
  3. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 204
  4. 2022Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisonscited by 170
  5. 2024The Impact of Reasoning Step Length on Large Language Modelscited by 153
  6. 2024Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Codecited by 107
  7. 2024Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Eracited by 98
  8. 2023Temporal Data Meets LLM -- Explainable Financial Time Series Forecastingcited by 77
  9. 2023Is ChatGPT the Ultimate Programming Assistant -- How far is it?cited by 76
  10. 2022Enhancing Self-Consistency and Performance of Pre-Trained Language Models through Natural Language Inferencecited by 40
  11. 2020SCNet: Training Inference Sample Consistency for Instance Segmentationcited by 38
  12. 2024LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Acceleratorscited by 32
  13. 2024LLM-Generated Black-box Explanations Can Be Adversarially Helpfulcited by 21
  14. 2024Fine-tuning Large Language Models for Entity Matchingcited by 16
  15. 2023Benchmarking and Explaining Large Language Model-based Code Generation: A Causality-Centric Approachcited by 15
  16. 2022A Simple, Yet Effective Approach to Finding Biases in Code Generationcited by 8
  17. 2022Automatically Recommend Code Updates: Are We There Yet?cited by 7
  18. 2024New Solutions on LLM Acceleration, Optimization, and Applicationcited by 7
  19. 2024Can GPT-O1 Kill All Bugs? An Evaluation of GPT-Family LLMs on QuixBugscited by 7
  20. 2022Quantifying Relevance in Learning and Inferencecited by 5
  21. 2022A Survey of Causal Inference Frameworkscited by 5
  22. 2024UELLM: A Unified and Efficient Approach for LLM Inference Servingcited by 2

Built on by

Not cited yet.