# NeuroSketch: A Practical Design Recipe for Neural Decoding

arXiv: https://arxiv.org/abs/2512.09524 · HTML: https://mlsanity.com/paper/2512.09524/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 74 references that are in the map.
- First posted on arXiv 10 Dec 2025.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Designing Network Design Spaces](https://mlsanity.com/paper/2003.13678/) (2020)
- [iTransformer: Inverted Transformers Are Effective for Time Series Forecasting](https://mlsanity.com/paper/2310.06625/) (2023)
- [EEGNet: A Compact Convolutional Network for EEG-based Brain-Computer Interfaces](https://mlsanity.com/paper/1611.08024/) (2016)
- [Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks](https://mlsanity.com/paper/1910.01279/) (2019)
- [MetaFormer Is Actually What You Need for Vision](https://mlsanity.com/paper/2111.11418/) (2021)
- [CBraMod: A Criss-Cross Brain Foundation Model for EEG Decoding](https://mlsanity.com/paper/2412.07236/) (2024)
- [MetaFormer Baselines for Vision](https://mlsanity.com/paper/2210.13452/) (2022)
- [TimeMixer++: A General Time Series Pattern Machine for Universal Predictive Analysis](https://mlsanity.com/paper/2410.16032/) (2024)
- [MultiGrain: a unified image embedding for classes and instances](https://mlsanity.com/paper/1902.05509/) (2019)
- [Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision](https://mlsanity.com/paper/1608.03425/) (2016)
- [Medformer: A Multi-Granularity Patching Transformer for Medical Time-Series Classification](https://mlsanity.com/paper/2405.19363/) (2024)
- [Squeezeformer: An Efficient Transformer for Automatic Speech Recognition](https://mlsanity.com/paper/2206.00888/) (2022)
- [Automatic Sleep Staging of EEG Signals: Recent Development, Challenges, and Future Directions](https://mlsanity.com/paper/2111.08446/) (2021)
- [Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groups](https://mlsanity.com/paper/1605.06489/) (2016)
- [EEG2IMAGE: Image Reconstruction from EEG Brain Signals](https://mlsanity.com/paper/2302.10121/) (2023)
- [Neural decoding from stereotactic EEG: accounting for electrode variability across subjects](https://mlsanity.com/paper/2411.10458/) (2024)
- [Du-IN: Discrete units-guided mask modeling for decoding speech from Intracranial Neural signals](https://mlsanity.com/paper/2405.11459/) (2024)
- [Understanding Transformers for Time Series: Rank Structure, Flow-of-ranks, and Compressibility](https://mlsanity.com/paper/2510.03358/) (2025)
- [GuessTheMusic: Song Identification from Electroencephalography response](https://mlsanity.com/paper/2009.08793/) (2020)
- [Why Attention Fails: The Degeneration of Transformers into MLPs in Time Series Forecasting](https://mlsanity.com/paper/2509.20942/) (2025)

## Abstract (arXiv)

Neural decoding is fundamental to brain-computer interfaces, with growing applications in healthcare. Previous research has focused on leveraging signal processing and deep learning methods to enhance neural decoding performance. However, systematic guidance on architectural design for neural decoding remains limited. In this study, we develop NeuroSketch, a practical design recipe for neural decoding, through a basic architecture study followed by macro- and micro-level optimization. Comparing nine basic architectures, we find that CNN-2D outperforms other architectures in neural decoding tasks and explore its effectiveness from temporal and spatial perspectives. Building on this backbone, we combine gradual feature-map expansion and early downsampling at the macro level with grouped convolutions at the micro level. These choices form the recipe, which we instantiate as NeuroSketch-Base (1.4M parameters) and NeuroSketch-Large (4.2M parameters). The recipe is developed and evaluated through nearly 5,000 experiments across eight tasks spanning visual, auditory, and speech modalities and EEG, SEEG, and ECoG signals. Against ten baselines, the two variants collectively achieve the best accuracy on each task. Our code is available at https://github.com/Galaxy-Dawn/NeuroSketch.