mlsanityData to 7 Oct 2026

Pico: A Modular Framework for Hypothesis-Driven Small Language Model Research

arXiv 2509.16413 ↗First posted 19 Sep 2025cs.CL cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Building language models (LMs), especially small and medium ones, remains more art than science. While large LMs often improve by sheer scale, it is still unclear why many design choices work. For small LMs, this uncertainty is more limiting: tight parameter budgets make each decision critical, yet researchers still lack systematic, scientific ways to test and refine new ideas. We introduce Pico, a lightweight, modular framework that enables systematic, hypothesis-driven research for small and medium-scale language model development. Pico consists of two libraries that together provide a practical sandbox where researchers can make targeted changes to a model's architecture or training procedures and directly observe their effects on the model's behavior. To support reproducible experimentation, we also release a suite of baseline models, pico-decoder, trained under standardized conditions and open-sourced for the community. Case studies highlight how Pico can support iterative small LM design and analysis.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 39 of its 51 references that are in the map. First posted on arXiv 19 Sep 2025.

Builds on

39 of 51 references.

References · 39

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  11. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
27 more
  1. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  2. 2020GLU Variants Improve Transformercited by 1,724
  3. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  4. 2019Root Mean Square Layer Normalizationcited by 1,425
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  6. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  7. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  8. 2019Are Sixteen Heads Really Better than One?cited by 803
  9. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  10. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  11. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  12. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  13. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  14. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  15. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  16. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  17. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  18. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  19. 2018Insights on representational similarity in neural networks with canonical correlationcited by 318
  20. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 200
  21. 2024A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthinesscited by 125
  22. 2023Paloma: A Benchmark for Evaluating Language Model Fitcited by 53
  23. 2020Self-Supervised Meta-Learning for Few-Shot Natural Language Classification Taskscited by 51
  24. 2023Emergent inabilities? Inverse scaling over the course of pretrainingcited by 7
  25. 2025Learning Dynamics of Meta-Learning in Small Model Pretrainingcited by 3
  26. 2025Meta-Pretraining for Zero-Shot Cross-Lingual Named Entity Recognition in Low-Resource Philippine Languagescited by 3
  27. 2023Large Language Models Sometimes Generate Purely Negatively-Reinforced Textcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Vocabulary shapes cross-lingual variation of word-order learnability in language modelscited by 0
  2. 2026OpenLanguageModel: Readable and Composable Small-Language-Model Pretraining for Education and Researchcited by 0
  3. 2026Beetle: A Bilingual Model Suite for Modelling Second-Language Processingcited by 0