mlsanityData to 14 Oct 2026

NNsight and NDIF: Democratizing Access to Open-Weight Foundation Model Internals

arXiv 2407.14561 ↗First posted 18 Jul 2024cs.LG cs.AIcited by 45

Not on the map. Why?

Abstract (arXiv)

We introduce NNsight and NDIF, technologies that work in tandem to enable scientific study of the representations and computations learned by very large neural networks. NNsight is an open-source system that extends PyTorch to introduce deferred remote execution. The National Deep Inference Fabric (NDIF) is a scalable inference service that executes NNsight requests, allowing users to share GPU resources and pretrained models. These technologies are enabled by the Intervention Graph, an architecture developed to decouple experimental design from model runtime. Together, this framework provides transparent and efficient access to the internals of deep neural networks such as very large language models (LLMs) without imposing the cost or complexity of hosting customized models individually. We conduct a quantitative survey of the machine learning literature that reveals a growing gap in the study of the internals of large-scale AI. We demonstrate the design and use of our framework to address this gap by enabling a range of research methods on huge models. Finally, we conduct benchmarks to compare performance with previous approaches. Code, documentation, and tutorials are available at https://nnsight.net/.

Key facts. Cited by 45 papers in the map (data through 14 Oct 2026). Builds on 35 of its 54 references that are in the map. First posted on arXiv 18 Jul 2024.

Builds on

35 of 54 references.

References · 35

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  10. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  11. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  12. 2022Emergent Abilities of Large Language Modelscited by 2,512
23 more
  1. 2024Qwen2 Technical Reportcited by 2,120
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  3. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  4. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  5. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  6. 2024Yi: Open Foundation Models by 01.AIcited by 719
  7. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 708
  8. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  9. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
  10. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 523
  11. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 435
  12. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  13. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 359
  14. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 126
  15. 2024AtP*: An efficient and scalable method for localizing LLM behaviour to componentscited by 111
  16. 2023Interpretability at Scale: Identifying Causal Mechanisms in Alpacacited by 103
  17. 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 87
  18. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 85
  19. 2023Punica: Multi-Tenant LoRA Servingcited by 83
  20. 2021"Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classificationcited by 66
  21. 2024pyvene: A Library for Understanding and Improving PyTorch Models via Interventionscited by 54
  22. 2022Structured access: an emerging paradigm for safe AI deploymentcited by 41
  23. 2023FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUscited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 13; 2026: 29.

45 papers.

Sort:

By area:

  1. 2024InterPLM: Discovering Interpretable Features in Protein Language Models via Sparse Autoencoderscited by 66
  2. 2024Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformerscited by 44
  3. 2025Language Models use Lookbacks to Track Beliefscited by 39
  4. 2024The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysiscited by 24
  5. 2025Back Attention: Understanding and Enhancing Multi-Hop Reasoning in Large Language Modelscited by 23
  6. 2025Localized Cultural Knowledge is Conserved and Controllable in Large Language Modelscited by 20
  7. 2025Do Activation Verbalization Methods Convey Privileged Information?cited by 10
  8. 2025When and How Does CLIP Enable Domain and Compositional Generalization?cited by 9
  9. 2026Learning a Generative Meta-Model of LLM Activationscited by 8
  10. 2026The Truthfulness Spectrum Hypothesiscited by 8
  11. 2026Interpretability Can Be Actionablecited by 7
  12. 2026Learning State-Tracking from Code Using Linear RNNscited by 6
33 more
  1. 2025In-Context Algebracited by 5
  2. 2026TADA! Tuning Audio Diffusion Models through Activation Steeringcited by 4
  3. 2025Signatures of human-like processing in Transformer forward passescited by 3
  4. 2026Vision-Default, Prior-Override: Causal Mechanisms of Perception-Knowledge Conflict in Vision-Language Modelscited by 3
  5. 2026PyHealth 2.0: A Comprehensive Open-Source Toolkit for Accessible and Reproducible Clinical Deep Learningcited by 2
  6. 2026How Open Must Language Models be to Enable Reliable Scientific Inference?cited by 2
  7. 2026Language-Switching Triggers Take a Latent Detour Through Language Modelscited by 2
  8. 2025eDIF: A European Deep Inference Fabric for Remote Interpretability of LLMcited by 1
  9. 2025Epistemic Familiarity is Associated With Belief Stability in Large Language Modelscited by 1
  10. 2025Constructive Circuit Amplification: Improving Math Reasoning in LLMs via Targeted Sub-Network Updatescited by 1
  11. 2025Measuring Mechanistic Independence: Can Bias Be Removed Without Erasing Demographics?cited by 1
  12. 2026Patches of Nonlinearity: Instruction Vectors in Large Language Modelscited by 1
  13. 2026The GRADIEND Python Package: An End-to-End System for Gradient-Based Feature Learningcited by 1
  14. 2026One Lens, Many Worlds : A Capability-Typed Interface for World-Model Interpretabilitycited by 1
  15. 2026Revealing Hidden Model Behaviors with Task-Specific Self-Reportscited by 1
  16. 2026Demystifying Entropy-based Selection for Chain-of-Thought Compression in Large Reasoning Modelscited by 1
  17. 2026Decoy Direction Optimization: A Post-Hoc Defense Against LLM Abliterationcited by 1
  18. 2026Readable Before Actionable: Causal Tracing of Indirect Prompt Injectioncited by 1
  19. 2025TDHook: A Lightweight Framework for Interpretabilitycited by 0
  20. 2025In-Context Learning Without Copyingcited by 0
  21. 2026Explaining the Explainer: Understanding the Inner Workings of Transformer-based Symbolic Regression Modelscited by 0
  22. 2026Fluid Reasoning Representationscited by 0
  23. 2026DreamReader: An Interpretability Toolkit for Text-to-Image Modelscited by 0
  24. 2026BrainSurgery: Reproducible and Reliable Declarative Weight Manipulations for Model Editing and Upcyclingcited by 0
  25. 2026VASAE: Naming SAE Dictionary Directions with Vocabulary-Aligned Anchoringcited by 0
  26. 2026Unboxing Diffusion Models for the Arts: Interactive Model Bending and Practice-Based Explainabilitycited by 0
  27. 2026TAGTorch: A PyTorch Library for Geometry, Topology, and Symmetry-Aware Machine Learningcited by 0
  28. 2026Activation Probes Surface Code-Security Signals that the Model's Output Missescited by 0
  29. 2026Mechanistic Interpretability of Structure-Aware Numerical Reasoning in LLaMA 3.1 8Bcited by 0
  30. 2026A Unifying Perspective on Language Model Representations: From Filler-Role Structure to Mechanistic Interpretabilitycited by 0
  31. 2026The Interlingua Hypothesis: LLMs Translate via a Latent Task-agnostic Feature Spacecited by 0
  32. 2026Limitations of Automated Simulatability: LLM Simulators Can Bypass Explanationscited by 0
  33. 2026FLEET: From Logits Entropy to Enhanced Trajectories in Text Generationcited by 0