# General Information Metrics for Improving AI Model Training Efficiency

arXiv: https://arxiv.org/abs/2501.02004 · HTML: https://mlsanity.com/paper/2501.02004/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 74 references that are in the map.
- First posted on arXiv 2 Jan 2025.

## Builds on (most cited first, 30 of 32)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [DeepFM: A Factorization-Machine based Neural Network for CTR Prediction](https://mlsanity.com/paper/1703.04247/) (2017)
- [Textbooks Are All You Need](https://mlsanity.com/paper/2306.11644/) (2023)
- [Textbooks Are All You Need II: phi-1.5 technical report](https://mlsanity.com/paper/2309.05463/) (2023)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [Large Language Models: A Survey](https://mlsanity.com/paper/2402.06196/) (2024)
- [Learning Loss for Active Learning](https://mlsanity.com/paper/1905.03677/) (2019)
- [A Survey on Multimodal Large Language Models for Autonomous Driving](https://mlsanity.com/paper/2311.12320/) (2023)
- [Data-centric Artificial Intelligence: A Survey](https://mlsanity.com/paper/2303.10158/) (2023)
- [Conditional Time Series Forecasting with Convolutional Neural Networks](https://mlsanity.com/paper/1703.04691/) (2017)
- [Generative AI](https://mlsanity.com/paper/2309.07930/) (2023)
- [MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models](https://mlsanity.com/paper/2406.06046/) (2024)
- [A Comparative Survey of Deep Active Learning](https://mlsanity.com/paper/2203.13450/) (2022)
- [A Survey on Legal Judgment Prediction: Datasets, Metrics, Models and Challenges](https://mlsanity.com/paper/2204.04859/) (2022)
- [Towards Automated Neural Interaction Discovery for Click-Through Rate Prediction](https://mlsanity.com/paper/2007.06434/) (2020)
- [modAL: A modular active learning framework for Python](https://mlsanity.com/paper/1805.00979/) (2018)
- [Harnessing Diversity for Important Data Selection in Pretraining Large Language Models](https://mlsanity.com/paper/2409.16986/) (2024)
- [Click-Through Rate Prediction in Online Advertising: A Literature Review](https://mlsanity.com/paper/2202.10462/) (2022)
- [Data-Centric Artificial Intelligence](https://mlsanity.com/paper/2212.11854/) (2022)
- [A Data-Centric Approach for Training Deep Neural Networks with Less Data](https://mlsanity.com/paper/2110.03613/) (2021)
- [Repeated Random Sampling for Minimizing the Time-to-Accuracy of Learning](https://mlsanity.com/paper/2305.18424/) (2023)

## Built on by (most cited first, 4 of 4)

- [Information Science Principles of Machine Learning: A Causal Chain Meta-Framework Based on Formalized Information Mapping](https://mlsanity.com/paper/2505.13182/) (2025)
- [The COTe score: A decomposable framework for evaluating Document Layout Analysis models](https://mlsanity.com/paper/2603.12718/) (2026)
- [Information Physics of Intelligence: Unifying Logical Depth and Entropy under Thermodynamic Constraints](https://mlsanity.com/paper/2511.19156/) (2025)
- [Rethinking Calibration for High-Sparsity LLM Pruning: From Single Sources to Capability Coverage](https://mlsanity.com/paper/2606.03328/) (2026)

## Abstract (arXiv)

To address the growing size of AI model training data and the lack of a universal data selection methodology-factors that significantly drive up training costs -- this paper presents the General Information Metrics Evaluation (GIME) method. GIME leverages general information metrics from Objective Information Theory (OIT), including volume, delay, scope, granularity, variety, duration, sampling rate, aggregation, coverage, distortion, and mismatch to optimize dataset selection for training purposes. Comprehensive experiments conducted across diverse domains, such as CTR Prediction, Civil Case Prediction, and Weather Forecasting, demonstrate that GIME effectively preserves model performance while substantially reducing both training time and costs. Additionally, applying GIME within the Judicial AI Program led to a remarkable 39.56% reduction in total model training expenses, underscoring its potential to support efficient and sustainable AI development.