# Is Bigger Edit Batch Size Always Better? -- An Empirical Study on Model Editing with Llama-3

arXiv: https://arxiv.org/abs/2405.00664 · HTML: https://mlsanity.com/paper/2405.00664/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 11 references that are in the map.
- First posted on arXiv 1 May 2024.

## Builds on (most cited first, 8 of 8)

- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [Editing Large Language Models: Problems, Methods, and Opportunities](https://mlsanity.com/paper/2305.13172/) (2023)
- [Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models](https://mlsanity.com/paper/2301.04213/) (2023)
- [Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias](https://mlsanity.com/paper/2004.12265/) (2020)
- [A Brief Review of Hypernetworks in Deep Learning](https://mlsanity.com/paper/2306.06955/) (2023)
- [Model Editing at Scale leads to Gradual and Catastrophic Forgetting](https://mlsanity.com/paper/2401.07453/) (2024)
- [A Unified Framework for Model Editing](https://mlsanity.com/paper/2403.14236/) (2024)

## Built on by (most cited first, 8 of 8)

- [Representation Shattering in Transformers: A Synthetic Study with Knowledge Editing](https://mlsanity.com/paper/2410.17194/) (2024)
- [Lifelong Knowledge Editing requires Better Regularization](https://mlsanity.com/paper/2502.01636/) (2025)
- [Context-Robust Knowledge Editing for Language Models](https://mlsanity.com/paper/2505.23026/) (2025)
- [BTBR: A Bayesian-Theory-Driven Probabilistic-Fuzzy Framework for Implicit Bias Removal in Large Language Models](https://mlsanity.com/paper/2408.10608/) (2024)
- [Keys to Robust Edits: from Theoretical Insights to Practical Advances](https://mlsanity.com/paper/2410.09338/) (2024)
- [Efficient Knowledge Editing via Minimal Precomputation](https://mlsanity.com/paper/2506.04226/) (2025)
- [SHRED: Retain-Set-Free Unlearning via Self-Distillation with Logit Demotion](https://mlsanity.com/paper/2605.07482/) (2026)
- [Synthetic Worlds for Temporal Evaluation and Knowledge Updating in LLMs](https://mlsanity.com/paper/2609.00184/) (2026)

## Abstract (arXiv)

This study presents a targeted model editing analysis focused on the latest large language model, Llama-3. We explore the efficacy of popular model editing techniques - ROME, MEMIT, and EMMET, which are designed for precise layer interventions. We identify the most effective layers for targeted edits through an evaluation that encompasses up to 4096 edits across three distinct strategies: sequential editing, batch editing, and a hybrid approach we call as sequential-batch editing. Our findings indicate that increasing edit batch-sizes may degrade model performance more significantly than using smaller edit batches sequentially for equal number of edits. With this, we argue that sequential model editing is an important component for scaling model editing methods and future research should focus on methods that combine both batched and sequential editing. This observation suggests a potential limitation in current model editing methods which push towards bigger edit batch sizes, and we hope it paves way for future investigations into optimizing batch sizes and model editing performance.