mlsanityData to 7 Oct 2026

minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Models

arXiv 2203.13112 ↗First posted 24 Mar 2022cs.CLcited by 58

Not on the map. Why?

Abstract (arXiv)

We present minicons, an open source library that provides a standard API for researchers interested in conducting behavioral and representational analyses of transformer-based language models (LMs). Specifically, minicons enables researchers to apply analysis methods at two levels: (1) at the prediction level -- by providing functions to efficiently extract word/sentence level probabilities; and (2) at the representational level -- by also facilitating efficient extraction of word/phrase level vectors from one or more layers. In this paper, we describe the library and apply it to two motivating case studies: One focusing on the learning dynamics of the BERT architecture on relative grammatical judgments, and the other on benchmarking 23 different LMs on zero-shot abductive reasoning. minicons is available at https://github.com/kanishkamisra/minicons

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Builds on 36 of its 62 references that are in the map. First posted on arXiv 24 Mar 2022.

Builds on

36 of 62 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  8. 2019Language Models as Knowledge Bases?cited by 1,922
  9. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  10. 2018Neural Network Acceptability Judgmentscited by 952
  11. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  12. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
24 more
  1. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  2. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 350
  3. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  4. 2019Abductive Commonsense Reasoningcited by 300
  5. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  6. 2018Targeted Syntactic Evaluation of Language Modelscited by 239
  7. 2018A Simple Method for Commonsense Reasoningcited by 223
  8. 2020Unsupervised Commonsense Question Answering with Self-Talkcited by 161
  9. 2019Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic Statecited by 137
  10. 2020When Do You Need Billions of Words of Pretraining Data?cited by 104
  11. 2021The MultiBERTs: BERT Reproductions for Robustness Analysiscited by 89
  12. 2019ZuCo 2.0: A Dataset of Physiological Recordings During Natural Reading and Annotationcited by 65
  13. 2020Cross-Linguistic Syntactic Evaluation of Word Prediction Modelscited by 54
  14. 2020Human Sentence Processing: Recurrence or Attention?cited by 51
  15. 2020Exploring BERT's Sensitivity to Lexical Cues using Tests from Semantic Primingcited by 41
  16. 2021Refining Targeted Syntactic Evaluation of Language Modelscited by 41
  17. 2019Language Modelling Makes Sense: Propagating Representations through WordNet for Full-Coverage Word Sense Disambiguationcited by 40
  18. 2020jiant: A Software Toolkit for Research on General-Purpose Text Understanding Modelscited by 39
  19. 2019Blackbox meets blackbox: Representational Similarity and Stability Analysis of Neural Language Models and Brainscited by 37
  20. 2021Do language models learn typicality judgments from text?cited by 32
  21. 2019Analyzing and Interpreting Neural Networks for NLP: A Report on the First BlackboxNLP Workshopcited by 22
  22. 2020Contextualized Word Embeddings Encode Aspects of Human-Like Word Sense Knowledgecited by 18
  23. 2020diagNNose: A Library for Neural Activation Analysiscited by 7
  24. 2021Towards Zero-shot Commonsense Reasoning with Self-supervised Refinement of Language Modelscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 2; 2024: 19; 2025: 22; 2026: 11.

58 papers.

Sort:

By area:

  1. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  2. 2025Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 86
  3. 2024Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNscited by 65
  4. 2024Auxiliary task demands mask the capabilities of smaller language modelscited by 54
  5. 2024Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 48
  6. 2025Language Models Fail to Introspect About Their Knowledge of Languagecited by 28
  7. 2023A Better Way to Do Masked Language Model Scoringcited by 27
  8. 2022Language model acceptability judgements are not always robust to contextcited by 22
  9. 2024Teaching Models to Balance Resisting and Accepting Persuasioncited by 19
  10. 2025Both Direct and Indirect Evidence Contribute to Dative Alternation Preferences in Language Modelscited by 18
  11. 2022The Curious Case of Absolute Position Embeddingscited by 17
  12. 2024Robust Pronoun Fidelity with English LLMs: Are they Reasoning, Repeating, or Just Biased?cited by 17
38 more
  1. 2022COMPS: Conceptual Minimal Pair Sentences for testing Robust Property Knowledge and its Inheritance in Pre-trained Language Modelscited by 15
  2. 2024A Psycholinguistic Evaluation of Language Models' Sensitivity to Argument Rolescited by 14
  3. 2024Large Language Models are Capable of Offering Cognitive Reappraisal, if Guidedcited by 13
  4. 2024BEAR: A Unified Framework for Evaluating Relational Knowledge in Causal and Masked Language Modelscited by 11
  5. 2024Scope Ambiguities in Large Language Modelscited by 11
  6. 2025Are BabyLMs Deaf to Gricean Maxims? A Pragmatic Evaluation of Sample-efficient Language Modelscited by 11
  7. 2024Controlled Evaluation of Syntactic Knowledge in Multilingual Language Modelscited by 10
  8. 2024Characterizing the Role of Similarity in the Property Inferences of Language Modelscited by 9
  9. 2025Child-Directed Language Does Not Consistently Boost Syntax Learning in Language Modelscited by 9
  10. 2025TurBLiMP: A Turkish Benchmark of Linguistic Minimal Pairscited by 9
  11. 2024A systematic framework for generating novel experimental hypotheses from language modelscited by 8
  12. 2024Large Language Model Recall Uncertainty is Modulated by the Fan Effectcited by 7
  13. 2025Different types of syntactic agreement recruit the same units within large language modelscited by 7
  14. 2025Alignment Drift in CEFR-prompted LLMs for Interactive Spanish Tutoringcited by 6
  15. 2025Vision-and-Language Training Helps Deploy Taxonomic Knowledge but Does Not Fundamentally Alter Itcited by 6
  16. 2025HPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Modelscited by 6
  17. 2024Superlatives in Context: Modeling the Implicit Semantics of Superlativescited by 5
  18. 2024ProGRes: Prompted Generative Rescoring on ASR n-Bestcited by 5
  19. 2025Unpacking Let Alone: Human-Scale Models Generalize to a Rare Construction in Form but not Meaningcited by 4
  20. 2024Experimental Contexts Can Facilitate Robust Semantic Property Inference in Language Models, but Inconsistentlycited by 3
  21. 2024The Base-Rate Effect on LLM Benchmark Performance: Disambiguating Test-Taking Strategies from Benchmark Performancecited by 3
  22. 2025Is It JUST Semantics? A Case Study of Discourse Particle Understanding in LLMscited by 3
  23. 2022"No, they did not": Dialogue response dynamics in pre-trained language modelscited by 1
  24. 2024Models Can and Should Embrace the Communicative Nature of Human-Generated Mathcited by 1
  25. 2025Language Models Largely Exhibit Human-like Constituent Ordering Preferencescited by 1
  26. 2025All-in-one: Understanding and Generation in Multimodal Reasoning with the MAIA Benchmarkcited by 1
  27. 2025semantic-features: A User-Friendly Tool for Studying Contextual Word Embeddings in Interpretable Semantic Spacescited by 1
  28. 2025WUGNECTIVES: Novel Entity Inferences of Language Models from Discourse Connectivescited by 1
  29. 2025Hey, wait a minute: on at-issue sensitivity in Language Modelscited by 1
  30. 2025Dark & Stormy: Modeling Humor in Sentences from the Bulwer-Lytton Fiction Contestcited by 1
  31. 2025On the Interplay between Positional Encodings, Morphological Complexity, and Word Order Flexibilitycited by 1
  32. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1
  33. 2026A Unified Assessment of the Poverty of the Stimulus Argument for Neural Language Modelscited by 1
  34. 2026Perturbation: A simple and efficient adversarial tracer for representation learning in language modelscited by 1
  35. 2026GlossoGen: Emergent Language in Complex Multi-Agent LLM Interactionscited by 1
  36. 2024Code-switching in text and speech challenges information-theoretic speaker designcited by 0
  37. 2025SPACER: A Parallel Dataset of Speech Production And Comprehension of Error Repairscited by 0
  38. 2025Investigating Syntactic Biases in Multilingual Transformers with RC Attachment Ambiguities in Italian and Englishcited by 0

50 most cited of 58 All 58 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Transformer-Based Language Models Across Domain Verticals: Architectures, Applications and Critical Assessment
  2. 2023Jump to Conclusions: Short-Cutting Transformers With Linear Transformations
  3. 2025Speed Always Wins: A Survey on Efficient Architectures for Large Language Models
  4. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space
  5. 2024Does Representation Matter? Exploring Intermediate Layers in Large Language Models
  6. 2025SR-LLM: Rethinking the Structured Representation in Large Language Model
  7. 2025Modeling Language as a Sequence of Thoughts
  8. 2024Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformers