# NumPert: Numerical Perturbations to Probe Language Models for Veracity Prediction

arXiv: https://arxiv.org/abs/2511.09971 · HTML: https://mlsanity.com/paper/2511.09971/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 20 references that are in the map.
- First posted on arXiv 13 Nov 2025.

## Builds on (most cited first, 15 of 15)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models](https://mlsanity.com/paper/2503.16419/) (2025)
- [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021)
- [FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured information](https://mlsanity.com/paper/2106.05707/) (2021)
- [Do NLP Models Know Numbers? Probing Numeracy in Embeddings](https://mlsanity.com/paper/1909.07940/) (2019)
- [A Survey of Deep Learning for Mathematical Reasoning](https://mlsanity.com/paper/2212.10535/) (2022)
- [Number Cookbook: Number Understanding of Language Models and How to Improve It](https://mlsanity.com/paper/2411.03766/) (2024)
- [What Are the Odds? Language Models Are Capable of Probabilistic Reasoning](https://mlsanity.com/paper/2406.12830/) (2024)
- [QuanTemp: A real-world open-domain benchmark for fact-checking numerical claims](https://mlsanity.com/paper/2403.17169/) (2024)
- [Surprising Efficacy of Fine-Tuned Transformers for Fact-Checking over Larger Language Models](https://mlsanity.com/paper/2402.12147/) (2024)
- [Verifying the Robustness of Automatic Credibility Assessment](https://mlsanity.com/paper/2303.08032/) (2023)
- [Towards Robust Numerical Question Answering: Diagnosing Numerical Capabilities of NLP Systems](https://mlsanity.com/paper/2211.07455/) (2022)
- [Adversarial Attacks Against Automated Fact-Checking: A Survey](https://mlsanity.com/paper/2509.08463/) (2025)
- [MetaRuleGPT: Recursive Numerical Reasoning of Language Models Trained with Simple Rules](https://mlsanity.com/paper/2412.13536/) (2024)

## Built on by (most cited first, 1 of 1)

- [Towards Robust Numerical Claim Verification](https://mlsanity.com/paper/2610.00689/) (2026)

## Abstract (arXiv)

Large language models show strong performance on knowledge intensive tasks such as fact-checking and question answering, yet they often struggle with numerical reasoning. We present a systematic evaluation of state-of-the-art models for veracity prediction on numerical claims and evidence pairs using controlled perturbations, including label-flipping probes, to test robustness. Our results indicate that even leading proprietary systems experience accuracy drops of up to 62% under certain perturbations. No model proves to be robust across all conditions. We further find that increasing context length generally reduces accuracy, but when extended context is enriched with perturbed demonstrations, most models substantially recover. These findings highlight critical limitations in numerical fact-checking and suggest that robustness remains an open challenge for current language models.