# Red Teaming for Large Language Models At Scale: Tackling Hallucinations on Mathematics Tasks

arXiv: https://arxiv.org/abs/2401.00290 · HTML: https://mlsanity.com/paper/2401.00290/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security.
- Builds on 3 of its 17 references that are in the map.
- First posted on arXiv 30 Dec 2023.

## Placed in

- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 3 of 3)

- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [Assessing Language Model Deployment with Risk Cards](https://mlsanity.com/paper/2303.18190/) (2023)
- [A Puzzle-Based Dataset for Natural Language Inference](https://mlsanity.com/paper/2112.05742/) (2021)

## Built on by (most cited first, 3 of 3)

- [DiaHalu: A Dialogue-level Hallucination Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2403.00896/) (2024)
- [Recent Advances in Attack and Defense Approaches of Large Language Models](https://mlsanity.com/paper/2409.03274/) (2024)
- [EAIRA: Establishing a Methodology for Evaluating AI Models as Scientific Research Assistants](https://mlsanity.com/paper/2502.20309/) (2025)

## Abstract (arXiv)

We consider the problem of red teaming LLMs on elementary calculations and algebraic tasks to evaluate how various prompting techniques affect the quality of outputs. We present a framework to procedurally generate numerical questions and puzzles, and compare the results with and without the application of several red teaming techniques. Our findings suggest that even though structured reasoning and providing worked-out examples slow down the deterioration of the quality of answers, the gpt-3.5-turbo and gpt-4 models are not well suited for elementary calculations and reasoning tasks, also when being red teamed.