mlsanityData to 7 Oct 2026

A Theory of Usable Information Under Computational Constraints

arXiv 2002.10689 ↗First posted 25 Feb 2020cs.LG stat.MLcited by 153

Not on the map. Why?

Abstract (arXiv)

We propose a new framework for reasoning about information in complex systems. Our foundation is based on a variational extension of Shannon's information theory that takes into account the modeling power and computational constraints of the observer. The resulting emphpredictive V-information encompasses mutual information and other notions of informativeness such as the coefficient of determination. Unlike Shannon's mutual information and in violation of the data processing inequality, V-information can be created through computation. This is consistent with deep neural networks extracting hierarchies of progressively more informative features in representation learning. Additionally, we show that by incorporating computational constraints, V-information can be reliably estimated from data even in high dimensions with PAC-style guarantees. Empirically, we demonstrate predictive V-information is more effective than mutual information for structure learning and fair representation learning.

Key facts. Cited by 153 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 16; 2022: 15; 2023: 32; 2024: 32; 2025: 21; 2026: 37.

153 papers.

Sort:

By area:

  1. 2021Towards Out-Of-Distribution Generalization: A Surveycited by 431
  2. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  3. 2023LEACE: Perfect linear concept erasure in closed formcited by 254
  4. 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 194
  5. 2021Acquisition of Chess Knowledge in AlphaZerocited by 120
  6. 2023On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigmcited by 107
  7. 2023ReCEval: Evaluating Reasoning Chains via Correctness and Informativenesscited by 76
  8. 2023To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Reviewcited by 72
  9. 2024When Do We Not Need Larger Vision Models?cited by 65
  10. 2024How Large Language Models Encode Context Knowledge? A Layer-Wise Probing Studycited by 64
  11. 2021Conditional probing: measuring usable information beyond a baselinecited by 61
  12. 2022Probing for the Usage of Grammatical Numbercited by 57
38 more
  1. 2021Temporal Predictive Coding For Model-Based Planning In Latent Spacecited by 54
  2. 2025How much do language models memorize?cited by 53
  3. 2021Lossy Compression for Lossless Predictioncited by 51
  4. 2021What Context Features Can Transformer Language Models Use?cited by 49
  5. 2024Do language models plan ahead for future tokens?cited by 47
  6. 2021Controllable Guarantees for Fair Outcomes via Contrastive Information Estimationcited by 45
  7. 2023RePrompt: Automatic Prompt Editing to Refine AI-Generative Art Towards Precise Expressionscited by 43
  8. 2024PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrieverscited by 42
  9. 2023Disentanglement via Latent Quantizationcited by 40
  10. 2023Frozen Transformers in Language Models Are Effective Visual Encoder Layerscited by 39
  11. 2025On the generalization of language models from in-context learning and finetuning: a controlled studycited by 39
  12. 2021Fair Normalizing Flowscited by 38
  13. 2023FedML-HE: An Efficient Homomorphic-Encryption-Based Privacy-Preserving Federated Learning Systemcited by 38
  14. 2022Improving Self-Supervised Learning by Characterizing Idealized Representationscited by 33
  15. 2022Word Order Does Matter (And Shuffled Language Models Know It)cited by 32
  16. 2024On the Challenges and Opportunities in Generative AIcited by 31
  17. 2025Value Profiles for Encoding Human Variationcited by 29
  18. 2022On the Learning of Non-Autoregressive Transformerscited by 27
  19. 2023People Make Better Edits: Measuring the Efficacy of LLM-Generated Counterfactually Augmented Data for Harmful Language Detectioncited by 25
  20. 2023A Geometric Notion of Causal Probingcited by 23
  21. 2021A Bayesian Framework for Information-Theoretic Probingcited by 22
  22. 2022REV: Information-Theoretic Evaluation of Free-Text Rationalescited by 22
  23. 2022FARE: Provably Fair Representation Learning with Practical Certificatescited by 21
  24. 2025Lost in Latent Space: An Empirical Study of Latent Diffusion Models for Physics Emulationcited by 19
  25. 2021Tight Mutual Information Estimation With Contrastive Fenchel-Legendre Optimizationcited by 18
  26. 2024Arrows of Time for Large Language Modelscited by 18
  27. 2023VNE: An Effective Method for Improving Deep Representation by Manipulating Eigenvalue Distributioncited by 17
  28. 2024Understanding Visual Feature Reliance through the Lens of Complexitycited by 15
  29. 2022Critical Learning Periods for Multisensory Integration in Deep Networkscited by 14
  30. 2023Operationalising Representation in Natural Language Processingcited by 14
  31. 2024On Subjective Uncertainty Quantification and Calibration in Natural Language Generationcited by 14
  32. 2022Machine Learning with Confidential Computing: A Systematization of Knowledgecited by 13
  33. 2022Gacs-Korner Common Information Variational Autoencodercited by 12
  34. 2024One-shot Federated Learning via Synthetic Distiller-Distillate Communicationcited by 12
  35. 2025Aligning Multimodal Representations through an Information Bottleneckcited by 12
  36. 2023Beyond Distribution Shift: Spurious Features Through the Lens of Training Dynamicscited by 11
  37. 2023Gaussian Process Probes (GPP) for Uncertainty-Aware Probingcited by 11
  38. 2023Understanding Probe Behaviors through Variational Bounds of Mutual Informationcited by 11

50 most cited of 153 All 153 (JSON)