mlsanityData to 7 Oct 2026

Rissanen Data Analysis: Examining Dataset Characteristics via Description Length

arXiv 2103.03872 ↗First posted 5 Mar 2021cs.LG cs.AI cs.CLcited by 22

Not on the map. Why?

Abstract (arXiv)

We introduce a method to determine if a certain capability helps to achieve an accurate model of given data. We view labels as being generated from the inputs by a program composed of subroutines with different capabilities, and we posit that a subroutine is useful if and only if the minimal program that invokes it is shorter than the one that does not. Since minimum program length is uncomputable, we instead estimate the labels' minimum description length (MDL) as a proxy, giving us a theoretically-grounded method for analyzing dataset characteristics. We call the method Rissanen Data Analysis (RDA) after the father of MDL, and we showcase its applicability on a wide variety of settings in NLP, ranging from evaluating the utility of generating subquestions before answering a question, to analyzing the value of rationales and explanations, to investigating the importance of different parts of speech, and uncovering dataset gender bias.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 65 of its 111 references that are in the map. First posted on arXiv 5 Mar 2021.

Builds on

65 of 111 references.

References · 65

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2017On Calibration of Modern Neural Networkscited by 4,860
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  11. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  12. 2020Longformer: The Long-Document Transformercited by 3,362
38 more
  1. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  2. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  3. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  4. 2015VQA: Visual Question Answeringcited by 2,489
  5. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  6. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  7. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  8. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  9. 2018Datasheets for Datasetscited by 1,371
  10. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  11. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  12. 2017Mixed Precision Trainingcited by 1,135
  13. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  14. 2018Neural Network Acceptability Judgmentscited by 952
  15. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  16. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  17. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  18. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  19. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  20. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  21. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  22. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  23. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  24. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 423
  25. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  26. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  27. 2020Calibration of Pre-trained Transformerscited by 307
  28. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 291
  29. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 230
  30. 2020Information-Theoretic Probing for Linguistic Structurecited by 204
  31. 2020Measuring Association Between Labels and Free-Text Rationalescited by 159
  32. 2019Queens are Powerful too: Mitigating Gender Bias in Dialogue Generationcited by 152
  33. 2019Multi-hop Reading Comprehension through Question Decomposition and Rescoringcited by 119
  34. 2020Unsupervised Question Decomposition for Question Answeringcited by 118
  35. 2015Yin and Yang: Balancing and Answering Binary Visual Questionscited by 109
  36. 2020Evaluating Explanations: How much do explanations from the teacher aid students?cited by 94
  37. 2018Understanding the Origins of Bias in Word Embeddingscited by 90
  38. 2019Compositional Questions Do Not Necessitate Multi-hop Reasoningcited by 87

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year

Citing papers per year: 2021: 6; 2022: 4; 2023: 7; 2024: 4; 2025: 1.

22 papers.

Sort:

By area:

  1. 2021True Few-Shot Learning with Language Modelscited by 343
  2. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  3. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  4. 2021What Context Features Can Transformer Language Models Use?cited by 49
  5. 2024The Unreasonable Effectiveness of Easy Training Data for Hard Taskscited by 40
  6. 2021Rationales for Sequential Predictionscited by 26
  7. 2023Bridging Information-Theoretic and Geometric Compression in Language Modelscited by 26
  8. 2023Out-of-Distribution Detection by Leveraging Between-Layer Transformation Smoothnesscited by 15
  9. 2022Sequential Learning Of Neural Networks for Prequential MDLcited by 13
  10. 2024Plausibly Problematic Questions in Multiple-Choice Benchmarks for Commonsense Reasoningcited by 13
  11. 2022Which Shortcut Solution Do Question Answering Models Prefer to Learn?cited by 9
  12. 2023Data Similarity is Not Enough to Explain Language Model Performancecited by 9
10 more
  1. 2021Comparing Text Representations: A Theory-Driven Approachcited by 5
  2. 2022A Latent-Variable Model for Intrinsic Probingcited by 5
  3. 2025Revisiting Generalization Across Difficulty Levels: It's Not So Easycited by 5
  4. 2022On Measuring the Intrinsic Few-Shot Hardness of Datasetscited by 4
  5. 2024Identifying Task Groupings for Multi-Task Learning Using Pointwise V-Usable Informationcited by 4
  6. 2023On Dataset Transferability in Active Learning for Transformerscited by 3
  7. 2023Thrust: Adaptively Propels Large Language Models with External Knowledgecited by 3
  8. 2023The Map Equation Goes Neural: Mapping Network Flows with Graph Neural Networkscited by 2
  9. 2023Measuring Pointwise V-Usable Information In-Context-lycited by 0
  10. 2024textitGeoHard: Towards Measuring Class-wise Hardness through Modelling Class Semanticscited by 0