# RobuNFR: Evaluating the Robustness of Large Language Models on Non-Functional Requirements Aware Code Generation

arXiv: https://arxiv.org/abs/2503.22851 · HTML: https://mlsanity.com/paper/2503.22851/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 42 references that are in the map.
- First posted on arXiv 28 Mar 2025.

## Builds on (most cited first, 20 of 20)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Large Language Models: A Survey](https://mlsanity.com/paper/2402.06196/) (2024)
- [How is ChatGPT's behavior changing over time?](https://mlsanity.com/paper/2307.09009/) (2023)
- [ReCode: Robustness Evaluation of Code Generation Models](https://mlsanity.com/paper/2212.10264/) (2022)
- [CodeScore: Evaluating Code Generation by Learning Code Execution](https://mlsanity.com/paper/2301.09043/) (2023)
- [A Survey on Large Language Models for Software Engineering](https://mlsanity.com/paper/2312.15223/) (2023)
- [SOEN-101: Code Generation by Emulating Software Process Models Using Large Language Model Agents](https://mlsanity.com/paper/2403.15852/) (2024)
- [Search-Based LLMs for Code Optimization](https://mlsanity.com/paper/2408.12159/) (2024)
- [ECCO: Can We Improve Model-Generated Code Efficiency Without Sacrificing Functional Correctness?](https://mlsanity.com/paper/2407.14044/) (2024)
- [NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctness](https://mlsanity.com/paper/2401.15963/) (2024)
- [AI-powered Code Review with LLMs: Early Results](https://mlsanity.com/paper/2404.18496/) (2024)
- [NLPerturbator: Studying the Robustness of Code LLMs to Natural Language Variations](https://mlsanity.com/paper/2406.19783/) (2024)
- [Exploring the Robustness of Large Language Models for Solving Programming Problems](https://mlsanity.com/paper/2306.14583/) (2023)
- [MANTRA: Enhancing Automated Method-Level Refactoring with Contextual RAG and Multi-Agent LLM Collaboration](https://mlsanity.com/paper/2503.14340/) (2025)
- [Are They All Good? Studying Practitioners' Expectations on the Readability of Log Messages](https://mlsanity.com/paper/2308.08836/) (2023)
- [PACE: A Program Analysis Framework for Continuous Performance Prediction](https://mlsanity.com/paper/2312.00918/) (2023)
- [Revisiting Exception Handling Practices with Exception Flow Analysis](https://mlsanity.com/paper/1708.00817/) (2017)
- [Exploring Language Model's Code Generation Ability with Auxiliary Functions](https://mlsanity.com/paper/2403.10575/) (2024)
- [EasyView: Bringing Performance Profiles into Integrated Development Environments](https://mlsanity.com/paper/2312.16598/) (2023)

## Built on by (most cited first, 5 of 5)

- [Success is in the Details: Evaluate and Enhance Details Sensitivity of Code LLMs through Counterfactuals](https://mlsanity.com/paper/2505.14597/) (2025)
- [Instance-level Randomization: Toward More Stable LLM Evaluations](https://mlsanity.com/paper/2509.12678/) (2025)
- [BabelCoder: Agentic Code Translation with Specification Alignment](https://mlsanity.com/paper/2512.06902/) (2025)
- [Accuracy and Satisfaction in Multi-Turn LLM Dialogues for NFR Assessment](https://mlsanity.com/paper/2606.24834/) (2026)
- [Noisy Test-Time Reinforcement Learning for Code LLMs](https://mlsanity.com/paper/2609.32172/) (2026)

## Abstract (arXiv)

When using LLMs to address Non-Functional Requirements (NFRs), developers may behave differently (e.g., expressing the same NFR in different words). Robust LLMs should output consistent results across these variations; however, this aspect remains underexplored. We propose RobuNFR for evaluating the robustness of LLMs in NFR-aware code generation across four NFR dimensions: design, readability, reliability, and performance, using three methodologies: prompt variation, regression testing, and diverse workflows. Our experiments show that RobuNFR reveals robustness issues in the tested LLMs when considering NFRs in code generation. Specifically, under prompt variation, including NFRs leads to a decrease in Pass@1 by up to 39 percent and an increase in the standard deviation from 0.48 to 2.48 compared to the baseline without NFRs (i.e., Function-Only). While incorporating NFRs generally improves overall NFR metrics, it also results in higher prompt sensitivity. In regression settings, some LLMs exhibit differences across versions, with improvements in one aspect (e.g., reduced code smells) often accompanied by regressions in another (e.g., decreased correctness), revealing inconsistencies that challenge their robustness. When varying workflows, the tested LLMs show significantly different NFR-aware code generation capabilities between two workflows: (1) integrating NFRs and functional requirements into the initial prompt and (2) enhancing Function-Only-generated code with the same NFR.