# LLMs for Science: Usage for Code Generation and Data Analysis

arXiv: https://arxiv.org/abs/2311.16733 · HTML: https://mlsanity.com/paper/2311.16733/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 26 references that are in the map.
- First posted on arXiv 28 Nov 2023.

## Builds on (most cited first, 14 of 14)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation](https://mlsanity.com/paper/2211.11501/) (2022)
- [A Systematic Evaluation of Large Language Models of Code](https://mlsanity.com/paper/2202.13169/) (2022)
- [The Impact of AI on Developer Productivity: Evidence from GitHub Copilot](https://mlsanity.com/paper/2302.06590/) (2023)
- [The Diminishing Returns of Masked Language Models to Science](https://mlsanity.com/paper/2205.11342/) (2022)
- [L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models](https://mlsanity.com/paper/2309.17446/) (2023)

## Built on by (most cited first, 30 of 34)

- [A Complete Survey on LLM-based AI Chatbots](https://mlsanity.com/paper/2406.16937/) (2024)
- [SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretation](https://mlsanity.com/paper/2405.08807/) (2024)
- [How Far Are We From AGI: Are LLMs All We Need?](https://mlsanity.com/paper/2405.10313/) (2024)
- [Drawing Pandas: A Benchmark for LLMs in Generating Plotting Code](https://mlsanity.com/paper/2412.02764/) (2024)
- [A Survey on Large Language Model-based Agents for Statistics and Data Science](https://mlsanity.com/paper/2412.14222/) (2024)
- [Large Language Models for Computer-Aided Design: A Survey](https://mlsanity.com/paper/2505.08137/) (2025)
- [On Catastrophic Inheritance of Large Foundation Models](https://mlsanity.com/paper/2402.01909/) (2024)
- [InverseCoder: Self-improving Instruction-Tuned Code LLMs with Inverse-Instruct](https://mlsanity.com/paper/2407.05700/) (2024)
- [InferA: A Smart Assistant for Cosmological Ensemble Data](https://mlsanity.com/paper/2510.12920/) (2025)
- [OpInf-LLM: Parametric PDE Solving with LLMs via Operator Inference](https://mlsanity.com/paper/2602.01493/) (2026)
- [FhGenie: A Custom, Confidentiality-preserving Chat AI for Corporate and Scientific Use](https://mlsanity.com/paper/2403.00039/) (2024)
- [Knowing What Not to Do: Leverage Language Model Insights for Action Space Pruning in Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2405.16854/) (2024)
- [Engineering Systems for Data Analysis Using Interactive Structured Inductive Programming](https://mlsanity.com/paper/2503.14488/) (2025)
- [Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education](https://mlsanity.com/paper/2505.18220/) (2025)
- [AutoStreamPipe: LLM Assisted Automatic Generation of Data Stream Processing Pipelines](https://mlsanity.com/paper/2510.23408/) (2025)
- [When AI Persuades: Adversarial Explanation Attacks on Human Trust in AI-Assisted Decision Making](https://mlsanity.com/paper/2602.04003/) (2026)
- [CONCUR: Benchmarking LLMs for Concurrent Code Generation](https://mlsanity.com/paper/2603.03683/) (2026)
- [Arti-"fickle" Intelligence: Using LLMs as a Tool for Inference in the Political and Social Sciences](https://mlsanity.com/paper/2504.03822/) (2025)
- [CIAO - Code In Architecture Out - Automated Software Architecture Documentation with Large Language Models](https://mlsanity.com/paper/2604.08293/) (2026)
- [SLEGO: A Collaborative Data Analytics System with LLM Recommender for Diverse Users](https://mlsanity.com/paper/2406.11232/) (2024)
- [The why, what, and how of AI-based coding in scientific research](https://mlsanity.com/paper/2410.02156/) (2024)
- [From Scientific Texts to Verifiable Code: Automating the Process with Transformers](https://mlsanity.com/paper/2501.05252/) (2025)
- [CrossPL: Evaluating Large Language Models on Cross Programming Language Code Generation](https://mlsanity.com/paper/2507.19904/) (2025)
- [Evaluating Generative AI Tools for Personalized Offline Recommendations: A Comparative Study](https://mlsanity.com/paper/2508.03710/) (2025)
- [Knowledge-Guided Multi-Agent Framework for Application-Level Software Code Generation](https://mlsanity.com/paper/2510.19868/) (2025)
- [AutoICE: Automatically Synthesizing Verifiable C Code via LLM-driven Evolution](https://mlsanity.com/paper/2512.07501/) (2025)
- [CogMCTS: A Novel Cognitive-Guided Monte Carlo Tree Search Framework for Iterative Heuristic Evolution with Large Language Models](https://mlsanity.com/paper/2512.08609/) (2025)
- [Conversational AI for Rapid Scientific Prototyping: A Case Study on ESA's ELOPE Competition](https://mlsanity.com/paper/2601.04920/) (2026)
- [Towards automated data analysis: A guided framework for LLM-based risk estimation](https://mlsanity.com/paper/2603.04631/) (2026)
- [AblateCell: A Reproduce-then-Ablate Agent for Virtual Cell Repositories](https://mlsanity.com/paper/2604.19606/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have been touted to enable increased productivity in many areas of today's work life. Scientific research as an area of work is no exception: the potential of LLM-based tools to assist in the daily work of scientists has become a highly discussed topic across disciplines. However, we are only at the very onset of this subject of study. It is still unclear how the potential of LLMs will materialise in research practice. With this study, we give first empirical evidence on the use of LLMs in the research process. We have investigated a set of use cases for LLM-based tools in scientific research, and conducted a first study to assess to which degree current tools are helpful. In this paper we report specifically on use cases related to software engineering, such as generating application code and developing scripts for data analytics. While we studied seemingly simple use cases, results across tools differ significantly. Our results highlight the promise of LLM-based tools in general, yet we also observe various issues, particularly regarding the integrity of the output these tools provide.