mlsanityData to 14 Oct 2026

Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models

arXiv 2506.07247 ↗First posted 8 Jun 2025cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

We introduce Interactive Bayesian Distributional Robustness (IBDR), a novel Bayesian inference framework that allows modeling the interactions between particles, thereby enhancing ensemble quality through increased particle diversity. IBDR is grounded in a generalized theoretical framework that connects the distributional population loss with the approximate posterior, motivating a practical dual optimization procedure that enforces distributional robustness while fostering particle diversity. We evaluate IBDR's performance against various baseline methods using the VTAB-1K benchmark and the common reasoning language task. The results consistently show that IBDR outperforms these baselines, underscoring its effectiveness in real-world applications.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 58 of its 76 references that are in the map. First posted on arXiv 8 Jun 2025.

Builds on

58 of 76 references.

References · 58

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  6. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  10. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  11. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  12. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
38 more
  1. 2017Deep Mutual Learningcited by 628
  2. 2017Exploring Generalization in Deep Learningcited by 600
  3. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  4. 2016Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithmcited by 507
  5. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 497
  6. 2015Variational Dropout and the Local Reparameterization Trickcited by 490
  7. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  8. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 464
  9. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 434
  10. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 425
  11. 2014Stochastic Gradient Hamiltonian Monte Carlocited by 343
  12. 2017Certifying Some Distributional Robustness with Principled Adversarial Trainingcited by 337
  13. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 324
  14. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 320
  15. 2017Three Factors Influencing Minima in SGDcited by 273
  16. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 260
  17. 2019Improving Adversarial Robustness via Promoting Ensemble Diversitycited by 213
  18. 2016Distributionally Robust Stochastic Optimization with Wasserstein Distancecited by 185
  19. 2020Efficient and Scalable Bayesian Neural Nets with Rank-1 Factorscited by 171
  20. 2017Multiplicative Normalizing Flows for Variational Bayesian Neural Networkscited by 157
  21. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
  22. 2015On the properties of variational approximations of Gibbs posteriorscited by 121
  23. 2023Bayesian Low-rank Adaptation for Large Language Modelscited by 115
  24. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  25. 2023Sharpness-Aware Gradient Matching for Domain Generalizationcited by 102
  26. 2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98
  27. 2020Relative Flatness and Generalizationcited by 88
  28. 2020The Implicit and Explicit Regularization Effects of Dropoutcited by 71
  29. 2022Sharp-MAML: Sharpness-Aware Model-Agnostic Meta Learningcited by 58
  30. 2024BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Modelscited by 50
  31. 2023Kronecker-Factored Approximate Curvature for Modern Neural Network Architecturescited by 49
  32. 2019Emergent properties of the local geometry of neural loss landscapescited by 46
  33. 2022SAM as an Optimal Relaxation of Bayescited by 41
  34. 2018Structured Variational Learning of Bayesian Neural Networks with Horseshoe Priorscited by 35
  35. 2023Flatness-Aware Minimization for Domain Generalizationcited by 27
  36. 2017Wasserstein Distributionally Robust Optimization and Variation Regularizationcited by 25
  37. 2021Bayesian Attention Belief Networkscited by 25
  38. 2024Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Modelscited by 20

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025DoRAN: Stabilizing Weight-Decomposed Low-Rank Adaptation via Noise Injection and Auxiliary Networkscited by 2
  2. 2025Hypernetwork-Driven Low-Rank Adaptation Across Attention Headscited by 1
  3. 2026Bayesian Adaptation Gym: A Benchmark for the Bayesian Low-Rank Adaptation of Multi-Modal Language Modelscited by 0