mlsanityData to 7 Oct 2026

MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworks

arXiv 2103.06132 ↗First posted 10 Mar 2021cs.LG cs.AI cs.CVcited by 32

Not on the map. Why?

Abstract (arXiv)

Recent strategies achieved ensembling "for free" by fitting concurrently diverse subnetworks inside a single base network. The main idea during training is that each subnetwork learns to classify only one of the multiple inputs simultaneously provided. However, the question of how to best mix these multiple inputs has not been studied so far. In this paper, we introduce MixMo, a new generalized framework for learning multi-input multi-output deep subnetworks. Our key motivation is to replace the suboptimal summing operation hidden in previous approaches by a more appropriate mixing mechanism. For that purpose, we draw inspiration from successful mixed sample data augmentations. We show that binary mixing in features - particularly with rectangular patches from CutMix - enhances results by making subnetworks stronger and more diverse. We improve state of the art for image classification on CIFAR-100 and Tiny ImageNet datasets. Our easy to implement models notably outperform data augmented deep ensembles, without the inference and memory overheads. As we operate in features and simply better leverage the expressiveness of large networks, we open a new line of research complementary to previous works.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 99 of its 139 references that are in the map. First posted on arXiv 10 Mar 2021.

Builds on

99 of 139 references.

References · 99

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  6. 2017On Calibration of Modern Neural Networkscited by 4,860
  7. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  8. 2014Going Deeper with Convolutionscited by 4,231
  9. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  10. 2016Wide Residual Networkscited by 3,057
  11. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  12. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
38 more
  1. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  2. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  3. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  5. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  6. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,390
  7. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  8. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  9. 2017Random Erasing Data Augmentationcited by 1,235
  10. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  11. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  12. 2019When Does Label Smoothing Help?cited by 985
  13. 2016Deep Variational Information Bottleneckcited by 915
  14. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 897
  15. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  16. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 809
  17. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  18. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  19. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
  20. 2016Deep Exploration via Bootstrapped DQNcited by 518
  21. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  22. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 461
  23. 2019Measuring Calibration in Deep Learningcited by 443
  24. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 438
  25. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 374
  26. 2017Snapshot Ensembles: Train 1, get M for freecited by 349
  27. 2019ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoringcited by 349
  28. 2018DropBlock: A regularization method for convolutional networkscited by 261
  29. 2020Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal Mixupcited by 260
  30. 2019Improving Adversarial Robustness via Promoting Ensemble Diversitycited by 213
  31. 2020ClassMix: Segmentation-Based Data Augmentation for Semi-Supervised Learningcited by 209
  32. 2020Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learningcited by 208
  33. 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
  34. 2020Proving the Lottery Ticket Hypothesis: Pruning is All You Needcited by 200
  35. 2020How Does Mixup Help With Robustness and Generalization?cited by 193
  36. 2020Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learningcited by 190
  37. 2018Knowledge Distillation by On-the-Fly Native Ensemblecited by 186
  38. 2019Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Visioncited by 182

50 most cited of 99 All 99 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 11; 2023: 10; 2024: 4; 2025: 1; 2026: 2.

32 papers.

Sort:

By area:

  1. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  2. 2021On the Practicality of Deterministic Epistemic Uncertaintycited by 62
  3. 2022Packed-Ensembles for Efficient Uncertainty Estimationcited by 49
  4. 2022A Survey of Mix-based Data Augmentation: Taxonomy, Methods, Applications, and Explainabilitycited by 30
  5. 2022FiLM-Ensemble: Probabilistic Deep Learning via Feature-wise Linear Modulationcited by 24
  6. 2023Image Data Augmentation Approaches: A Comprehensive Survey and Future directionscited by 23
  7. 2023DART: Diversify-Aggregate-Repeat Training Improves Generalization of Neural Networkscited by 20
  8. 2021Sparse MoEs meet Efficient Ensemblescited by 19
  9. 2022DataMUX: Data Multiplexing for Neural Networkscited by 18
  10. 2024A Survey on Mixup Augmentations and Beyondcited by 18
  11. 2021Tackling Catastrophic Forgetting and Background Shift in Continual Semantic Segmentationcited by 15
  12. 2022Learning Instance-Specific Augmentations by Capturing Local Invariancescited by 15
20 more
  1. 2023MIMONets: Multiple-Input-Multiple-Output Neural Networks Exploiting Computation in Superpositioncited by 14
  2. 2022Preventing Manifold Intrusion with Locality: Local Mixupcited by 11
  3. 2022DoubleMix: Simple Interpolation-Based Data Augmentation for Text Classificationcited by 10
  4. 2023Astroformer: More Data Might not be all you need for Classificationcited by 6
  5. 2022Learn2Weight: Parameter Adaptation against Similar-domain Adversarial Attackscited by 5
  6. 2023Tailoring Mixup to Data for Calibrationcited by 5
  7. 2022Towards efficient feature sharing in MIMO architecturescited by 4
  8. 2023MUX-PLMs: Data Multiplexing for High-throughput Language Modelscited by 4
  9. 2023Self-Supervised Representation Learning with Cross-Context Learning between Global and Hypercolumn Featurescited by 4
  10. 2024Enhanced Few-Shot Class-Incremental Learning via Ensemble Modelscited by 4
  11. 2023PruMUX: Augmenting Data Multiplexing with Model Compressioncited by 3
  12. 2021Robust Object Detection with Multi-input Multi-output Faster R-CNNcited by 1
  13. 2022Embedded Ensembles: Infinite Width Limit and Operating Regimescited by 1
  14. 2024RevMUX: Data Multiplexing with Reversible Adapters for Efficient LLM Batch Inferencecited by 1
  15. 2025Enhancing Ambiguous Dynamic Facial Expression Recognition with Soft Label-based Data Augmentationcited by 1
  16. 2026When Are Two Scores Better Than One? Investigating Ensembles of Diffusion Modelscited by 1
  17. 2026Uncertainty-Aware Rank-One MIMO Q Network Framework for Accelerated Offline Reinforcement Learningcited by 1
  18. 2023GoogLe2Net: Going Transverse with Convolutionscited by 0
  19. 2023Music Representing Corpus Virtual: An Open Sourced Library for Explorative Music Generation, Sound Design, and Instrument Creation with Artificial Intelligence and Machine Learningcited by 0
  20. 2024SAE: Single Architecture Ensemble Neural Networkscited by 0