# Aggregating empirical evidence from data strategy studies: a case on model quantization

arXiv: https://arxiv.org/abs/2505.00816 · HTML: https://mlsanity.com/paper/2505.00816/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 37 references that are in the map.
- First posted on arXiv 1 May 2025.

## Builds on (most cited first, 14 of 14)

- [Software Engineering for AI-Based Systems: A Survey](https://mlsanity.com/paper/2105.01984/) (2021)
- [A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification](https://mlsanity.com/paper/2205.07877/) (2022)
- [A Synthesis of Green Architectural Tactics for ML-Enabled Systems](https://mlsanity.com/paper/2312.09610/) (2023)
- [Successful Combination of Database Search and Snowballing for Identification of Primary Studies in Systematic Literature Studies](https://mlsanity.com/paper/2307.02612/) (2023)
- [Green My LLM: Studying the key factors affecting the energy consumption of code assistants](https://mlsanity.com/paper/2411.11892/) (2024)
- [A Framework for Using LLMs for Repository Mining Studies in Empirical Software Engineering](https://mlsanity.com/paper/2411.09974/) (2024)
- [Language Models in Software Development Tasks: An Experimental Analysis of Energy and Accuracy](https://mlsanity.com/paper/2412.00329/) (2024)
- [Energy-Efficient Respiratory Anomaly Detection in Premature Newborn Infants](https://mlsanity.com/paper/2202.10570/) (2022)
- [Software Artifact Mining in Software Engineering Conferences: A Meta-Analysis](https://mlsanity.com/paper/2207.08436/) (2022)
- [Verifiable and Energy Efficient Medical Image Analysis with Quantised Self-attentive Deep Neural Networks](https://mlsanity.com/paper/2209.15287/) (2022)
- [Impact of ML Optimization Tactics on Greener Pre-Trained ML Models](https://mlsanity.com/paper/2409.12878/) (2024)
- [Comparing Techniques for Aggregating Interrelated Replications in Software Engineering](https://mlsanity.com/paper/1807.06809/) (2018)
- [How Many Papers Should You Review? A Research Synthesis of Systematic Literature Reviews in Software Engineering](https://mlsanity.com/paper/2307.06056/) (2023)
- [Optimization Strategies for Enhancing Resource Efficiency in Transformers & Large Language Models](https://mlsanity.com/paper/2502.00046/) (2025)

## Built on by (most cited first, 1 of 1)

- [Does quantization affect models' performance on long-context tasks?](https://mlsanity.com/paper/2505.20276/) (2025)

## Abstract (arXiv)

Background: As empirical software engineering evolves, more studies adopt data strategies-approaches that investigate digital artifacts such as models, source code, or system logs rather than relying on human subjects. Synthesizing results from such studies introduces new methodological challenges. Aims: This study assesses the effects of model quantization on correctness and resource efficiency in deep learning (DL) systems. Additionally, it explores the methodological implications of aggregating evidence from empirical studies that adopt data strategies. Method: We conducted a research synthesis of six primary studies that empirically evaluate model quantization. We applied the Structured Synthesis Method (SSM) to aggregate the findings, which combines qualitative and quantitative evidence through diagrammatic modeling. A total of 19 evidence models were extracted and aggregated. Results: The aggregated evidence indicates that model quantization weakly negatively affects correctness metrics while consistently improving resource efficiency metrics, including storage size, inference latency, and GPU energy consumption-a manageable trade-off for many DL deployment contexts. Evidence across quantization techniques remains fragmented, underscoring the need for more focused empirical studies per technique. Conclusions: Model quantization offers substantial efficiency benefits with minor trade-offs in correctness, making it a suitable optimization strategy for resource-constrained environments. This study also demonstrates the feasibility of using SSM to synthesize findings from data strategy-based research.