# Robust Learning with the Hilbert-Schmidt Independence Criterion

arXiv: https://arxiv.org/abs/1910.00270 · HTML: https://mlsanity.com/paper/1910.00270/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Oct 2019.

## Built on by (most cited first, 30 of 36)

- [Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalization](https://mlsanity.com/paper/2106.06607/) (2021)
- [Independent mechanism analysis, a new concept?](https://mlsanity.com/paper/2106.05200/) (2021)
- [Improving Conditional Coverage via Orthogonal Quantile Regression](https://mlsanity.com/paper/2106.00394/) (2021)
- [Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts](https://mlsanity.com/paper/2407.09590/) (2024)
- [Quality-Agnostic Deepfake Detection with Intra-model Collaborative Learning](https://mlsanity.com/paper/2309.05911/) (2023)
- [Revisiting Hilbert-Schmidt Information Bottleneck for Adversarial Robustness](https://mlsanity.com/paper/2106.02734/) (2021)
- [Measuring Dependence with Matrix-based Entropy Functional](https://mlsanity.com/paper/2101.10160/) (2021)
- [Exploiting Independent Instruments: Identification and Distribution Generalization](https://mlsanity.com/paper/2202.01864/) (2022)
- [Uncovering Meanings of Embeddings via Partial Orthogonality](https://mlsanity.com/paper/2310.17611/) (2023)
- [Variance Covariance Regularization Enforces Pairwise Independence in Self-Supervised Representations](https://mlsanity.com/paper/2209.14905/) (2022)
- [Cauchy-Schwarz Divergence Information Bottleneck for Regression](https://mlsanity.com/paper/2404.17951/) (2024)
- [Functional Generalized Empirical Likelihood Estimation for Conditional Moment Restrictions](https://mlsanity.com/paper/2207.04771/) (2022)
- [Relaxed Quantile Regression: Prediction Intervals for Asymmetric Noise](https://mlsanity.com/paper/2406.03258/) (2024)
- [Model Compression for Domain Adaptation through Causal Effect Estimation](https://mlsanity.com/paper/2101.07086/) (2021)
- [Domain Adaptation with Cauchy-Schwarz Divergence](https://mlsanity.com/paper/2405.19978/) (2024)
- [Mining Stable Preferences: Adaptive Modality Decorrelation for Multimedia Recommendation](https://mlsanity.com/paper/2306.14179/) (2023)
- [Nonlinear Causal Discovery for Grouped Data](https://mlsanity.com/paper/2506.05120/) (2025)
- [Coping with Change: Learning Invariant and Minimum Sufficient Representations for Fine-Grained Visual Categorization](https://mlsanity.com/paper/2306.04893/) (2023)
- [Triple Disentangled Representation Learning for Multimodal Affective Analysis](https://mlsanity.com/paper/2401.16119/) (2024)
- [Deep Stable Representation Learning on Electronic Health Records](https://mlsanity.com/paper/2209.01321/) (2022)
- [A statistical approach to detect sensitive features in a group fairness setting](https://mlsanity.com/paper/2305.06994/) (2023)
- [Benchmarking Spiking Neural Network Learning Methods with Varying Locality](https://mlsanity.com/paper/2402.01782/) (2024)
- [MSSDF: Modality-Shared Self-supervised Distillation for High-Resolution Multi-modal Remote Sensing Image Learning](https://mlsanity.com/paper/2506.09327/) (2025)
- [Deconfounding Causal Inference through Two-Branch Framework with Early-Forking for Sensor-Based Cross-Domain Activity Recognition](https://mlsanity.com/paper/2507.03898/) (2025)
- [H-SPLID: HSIC-based Saliency Preserving Latent Information Decomposition](https://mlsanity.com/paper/2510.20627/) (2025)
- [Learning to Transfer with von Neumann Conditional Divergence](https://mlsanity.com/paper/2108.03531/) (2021)
- [Revisiting the Robustness of the Minimum Error Entropy Criterion: A Transfer Learning Case Study](https://mlsanity.com/paper/2307.08572/) (2023)
- [Bipartite Graph Variational Auto-Encoder with Fair Latent Representation to Account for Sampling Bias in Ecological Networks](https://mlsanity.com/paper/2403.02011/) (2024)
- [Understanding Token-level Topological Structures in Transformer-based Time Series Forecasting](https://mlsanity.com/paper/2404.10337/) (2024)
- [A Knowledge Enhanced Learning and Semantic Composition Model for Multi-Claim Fact Checking](https://mlsanity.com/paper/2104.13046/) (2021)

## Abstract (arXiv)

We investigate the use of a non-parametric independence measure, the Hilbert-Schmidt Independence Criterion (HSIC), as a loss-function for learning robust regression and classification models. This loss-function encourages learning models where the distribution of the residuals between the label and the model prediction is statistically independent of the distribution of the instances themselves. This loss-function was first proposed by Mooij et al. (2009) in the context of learning causal graphs. We adapt it to the task of learning for unsupervised covariate shift: learning on a source domain without access to any instances or labels from the unknown target domain, but with the assumption that p(y|x) (the conditional probability of labels given instances) remains the same in the target domain. We show that the proposed loss is expected to give rise to models that generalize well on a class of target domains characterised by the complexity of their description within a reproducing kernel Hilbert space. Experiments on unsupervised covariate shift tasks demonstrate that models learned with the proposed loss-function outperform models learned with standard loss functions, achieving state-of-the-art results on a challenging cell-microscopy unsupervised covariate shift task.