# LinBridge: A Learnable Framework for Interpreting Nonlinear Neural Encoding Models

arXiv: https://arxiv.org/abs/2410.20053 · HTML: https://mlsanity.com/paper/2410.20053/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 44 references that are in the map.
- First posted on arXiv 26 Oct 2024.

## Builds on (most cited first, 15 of 15)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Learnable latent embeddings for joint behavioral and neural analysis](https://mlsanity.com/paper/2204.00673/) (2022)
- [Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Brain's Ventral Visual Pathway](https://mlsanity.com/paper/1411.6422/) (2014)
- [Toward a realistic model of speech processing in the brain with self-supervised learning](https://mlsanity.com/paper/2206.01685/) (2022)
- [Economy Statistical Recurrent Units For Inferring Nonlinear Granger Causality](https://mlsanity.com/paper/1911.09879/) (2019)
- [Self-supervised models of audio effectively explain human cortical responses to speech](https://mlsanity.com/paper/2205.14252/) (2022)
- [Brain encoding models based on multimodal transformers can transfer across language and vision](https://mlsanity.com/paper/2305.12248/) (2023)
- [Neural Additive Vector Autoregression Models for Causal Discovery in Time Series](https://mlsanity.com/paper/2010.09429/) (2020)
- [Revealing Vision-Language Integration in the Brain with Multimodal Networks](https://mlsanity.com/paper/2406.14481/) (2024)
- [Coupling Artificial Neurons in BERT and Biological Neurons in the Human Brain](https://mlsanity.com/paper/2303.14871/) (2023)
- [Jacobian Regularizer-based Neural Granger Causality](https://mlsanity.com/paper/2405.08779/) (2024)
- [Connecting Neural Response measurements & Computational Models of language: a non-comprehensive guide](https://mlsanity.com/paper/2203.05300/) (2022)
- [A visual encoding model based on deep neural networks and transfer learning](https://mlsanity.com/paper/1902.08793/) (2019)

## Abstract (arXiv)

Neural encoding of artificial neural networks (ANNs) links their computational representations to brain responses, offering insights into how the brain processes information. Current studies mostly use linear encoding models for clarity, even though brain responses are often nonlinear. This has sparked interest in developing nonlinear encoding models that are still interpretable. To address this problem, we propose LinBridge, a learnable and flexible framework based on Jacobian analysis for interpreting nonlinear encoding models. LinBridge posits that the nonlinear mapping between ANN representations and neural responses can be factorized into a linear inherent component that approximates the complex nonlinear relationship, and a mapping bias that captures sample-selective nonlinearity. The Jacobian matrix, which reflects output change rates relative to input, enables the analysis of sample-selective mapping in nonlinear models. LinBridge employs a self-supervised learning strategy to extract both the linear inherent component and nonlinear mapping biases from the Jacobian matrices of the test set, allowing it to adapt effectively to various nonlinear encoding models. We validate the LinBridge framework in the scenario of neural visual encoding, using computational visual representations from CLIP-ViT to predict brain activity recorded via functional magnetic resonance imaging (fMRI). Our experimental results demonstrate that: 1) the linear inherent component extracted by LinBridge accurately reflects the complex mappings of nonlinear neural encoding models; 2) the sample-selective mapping bias elucidates the variability of nonlinearity across different levels of the visual processing hierarchy. This study presents a novel tool for interpreting nonlinear neural encoding models and offers fresh evidence about hierarchical nonlinearity distribution in the visual cortex.