# Evaluating Robustness of Large Language Models in Enterprise Applications: Benchmarks for Perturbation Consistency Across Formats and Languages

arXiv: https://arxiv.org/abs/2601.06341 · HTML: https://mlsanity.com/paper/2601.06341/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 12 references that are in the map.
- First posted on arXiv 9 Jan 2026.

## Builds on (most cited first, 8 of 8)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Are Large Language Models Really Robust to Word-Level Perturbations?](https://mlsanity.com/paper/2309.11166/) (2023)
- [StructEval: Benchmarking LLMs' Capabilities to Generate Structural Outputs](https://mlsanity.com/paper/2505.20139/) (2025)
- [NLPerturbator: Studying the Robustness of Code LLMs to Natural Language Variations](https://mlsanity.com/paper/2406.19783/) (2024)
- [Enhancing LLM Robustness to Perturbed Instructions: An Empirical Study](https://mlsanity.com/paper/2504.02733/) (2025)
- [Order Matters: Investigate the Position Bias in Multi-constraint Instruction Following](https://mlsanity.com/paper/2502.17204/) (2025)
- [Towards Understanding the Robustness of LLM-based Evaluations under Perturbations](https://mlsanity.com/paper/2412.09269/) (2024)
- [StructEval: Deepen and Broaden Large Language Model Assessment via Structured Evaluation](https://mlsanity.com/paper/2408.03281/) (2024)

## Built on by (most cited first, 5 of 5)

- [Fragile Thoughts: How Large Language Models Handle Chain-of-Thought Perturbations](https://mlsanity.com/paper/2603.03332/) (2026)
- [Mitigating Translationese Bias in Multilingual LLM-as-a-Judge via Disentangled Information Bottleneck](https://mlsanity.com/paper/2603.10351/) (2026)
- [Measuring Reasoning Quality in LLMs: A Multi-Dimensional Behavioral Framework](https://mlsanity.com/paper/2605.24661/) (2026)
- [Evaluating LLM-Based Grant Proposal Review via Structured Perturbations](https://mlsanity.com/paper/2603.08281/) (2026)
- [RAGStress: A controlled benchmark for evaluating retrieval-augmented generation under knowledge-base degradation](https://mlsanity.com/paper/2610.04691/) (2026)

## Abstract (arXiv)

Enterprise LLM applications require consistently high quality and reliable performance across diverse scenarios, demanding robustness to minor variations. Existing research shows that even small prompt changes can lead to substantial differences in output, but has mainly focused on a narrow set of perturbations with small academic datasets, limiting their relevance to real-world applications. To address this, we present a comprehensive benchmark suite that evaluates robustness across multiple perturbation types, including general text edits (e.g., punctuation, whitespace), formatting changes (e.g., JSON, YAML), multilingual and cross-lingual inputs, and positional variations in instructions. Evaluating 11 models ranging from 4B to 120B+ parameters, we find that minor perturbations reduce performance by up to 40 percentage points on key enterprise metrics. Critically, we demonstrate that the relationship between model size and robustness is more nuanced than conventional assumptions suggest: an 8B parameter model (Ministral 3 8B) outperforms most larger models, while another 8B model (Llama 3.1 8B) performs worst overall.