# Merit: Making every modality earn its contribution in multimodal attention decoding

arXiv: https://arxiv.org/abs/2610.17507 · HTML: https://mlsanity.com/paper/2610.17507/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 51 references that are in the map.
- First posted on arXiv 13 Oct 2026.

## Builds on (most cited first, 29 of 29)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning](https://mlsanity.com/paper/2506.09985/) (2025)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously](https://mlsanity.com/paper/1801.01489/) (2018)
- [What Makes Training Multi-Modal Classification Networks Hard?](https://mlsanity.com/paper/1905.12681/) (2019)
- [Balanced Multimodal Learning via On-the-fly Gradient Modulation](https://mlsanity.com/paper/2203.15332/) (2022)
- [RUBi: Reducing Unimodal Biases in Visual Question Answering](https://mlsanity.com/paper/1906.10169/) (2019)
- [Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)](https://mlsanity.com/paper/2203.12221/) (2022)
- [Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks](https://mlsanity.com/paper/2202.05306/) (2022)
- [Does my multimodal model learn cross-modal interactions? It's harder to tell than you might think!](https://mlsanity.com/paper/2010.06572/) (2020)
- [EEG-based Auditory Attention Decoding: Towards Neuro-Steered Hearing Devices](https://mlsanity.com/paper/2008.04569/) (2020)
- [Relating EEG to continuous speech using deep neural networks: a review](https://mlsanity.com/paper/2302.01736/) (2023)
- [DARNet: Dual Attention Refinement Network with Spatiotemporal Construction for Auditory Attention Detection](https://mlsanity.com/paper/2410.11181/) (2024)
- [An LSTM Based Architecture to Relate Speech Stimulus to EEG](https://mlsanity.com/paper/2002.10988/) (2020)
- [A DenseNet-based method for decoding auditory spatial attention with EEG](https://mlsanity.com/paper/2309.07690/) (2023)
- [NeuroHeed: Neuro-Steered Speaker Extraction using EEG Signals](https://mlsanity.com/paper/2307.14303/) (2023)
- [SWIM: Short-Window CNN Integrated with Mamba for EEG-Based Auditory Spatial Attention Decoding](https://mlsanity.com/paper/2409.19884/) (2024)
- [Linear stimulus reconstruction works on the KU Leuven audiovisual, gaze-controlled auditory attention decoding dataset](https://mlsanity.com/paper/2412.01401/) (2024)
- [ListenNet: A Lightweight Spatio-Temporal Enhancement Nested Network for Auditory Attention Detection](https://mlsanity.com/paper/2505.10348/) (2025)
- [Predicting speech intelligibility from EEG in a non-linear classification paradigm](https://mlsanity.com/paper/2105.06844/) (2021)
- [NEUROTOKEN: Joint Source and Directional AAD with Envelope Decoding via Conditional Flow Matching](https://mlsanity.com/paper/2610.00397/) (2026)

## Abstract (arXiv)

More than 1.5 billion people have hearing loss. When several people talk at once, hearing aids amplify every voice alike. Auditory attention decoding (AAD) aims to solve this by reading a listener's brain signals to find the attended talker, letting the device amplify only them. A real device will not see the brain alone; smart glasses and earbuds also track eyes, head, and scene. These behavioral signals help, as people often look at who they listen to. However, they show where a listener looks, not whom they hear. A decoder relying heavily on them fails when a listener cannot move their head or listens without looking - exactly when a brain-based device is needed. A multimodal decoder must use these signals without letting them replace the electroencephalogram (EEG). Accuracy alone cannot show whether it does. We present MERIT, a decoder and evaluation method asking: how much accuracy does each input earn? MERIT shuffles each input stream across test examples, measures the accuracy drop, and splits the result across streams using Shapley values. It also blocks two shortcuts: candidate audio is checked before training so it cannot reveal the answer, and the scoring function cannot beat chance if it ignores the brain. On a public dataset with EEG, eye tracking, head motion, and scene video, MERIT improves on the published benchmark, reaching 0.82 four-way accuracy within listeners and 0.74 on unseen listeners (chance is 0.25). The measurement reveals what accuracy hides: adding behavioral streams raises overall accuracy but almost halves the brain's share of it. On two other datasets, the same tests find a label imbalance in a widely used benchmark and flag results that accuracy alone would report as brain decoding.