# Efficient Conditionally Invariant Representation Learning

arXiv: https://arxiv.org/abs/2212.08645 · HTML: https://mlsanity.com/paper/2212.08645/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 44 references that are in the map.
- First posted on arXiv 16 Dec 2022.

## Builds on (most cited first, 17 of 17)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Domain Adaptation with Conditional Distribution Matching and Generalized Label Shift](https://mlsanity.com/paper/2003.04475/) (2020)
- [Sobolev Norm Learning Rates for Regularized Least-Squares Algorithm](https://mlsanity.com/paper/1702.07254/) (2017)
- [A Measure-Theoretic Approach to Kernel Conditional Mean Embeddings](https://mlsanity.com/paper/2002.03689/) (2020)
- [Model Patching: Closing the Subgroup Performance Gap with Data Augmentation](https://mlsanity.com/paper/2008.06775/) (2020)
- [Putting Fairness Principles into Practice: Challenges, Metrics, and Improvements](https://mlsanity.com/paper/1901.04562/) (2019)
- [Causally motivated Shortcut Removal Using Auxiliary Labels](https://mlsanity.com/paper/2105.06422/) (2021)
- [Optimal Rates for Regularized Conditional Mean Embedding Learning](https://mlsanity.com/paper/2208.01711/) (2022)
- [Out-of-distribution Generalization in the Presence of Nuisance-Induced Spurious Correlations](https://mlsanity.com/paper/2107.00520/) (2021)
- [Invariant and Transportable Representations for Anti-Causal Domain Shifts](https://mlsanity.com/paper/2207.01603/) (2022)
- [A Rigorous Theory of Conditional Mean Embeddings](https://mlsanity.com/paper/1912.00671/) (2019)
- [Nonparametric approximation of conditional expectation operators](https://mlsanity.com/paper/2012.12917/) (2020)
- [Kernel Partial Correlation Coefficient -- a Measure of Conditional Dependence](https://mlsanity.com/paper/2012.14804/) (2020)
- [Generalization Through The Lens Of Leave-One-Out Error](https://mlsanity.com/paper/2203.03443/) (2022)
- [Learning Counterfactually Invariant Predictors](https://mlsanity.com/paper/2207.09768/) (2022)

## Built on by (most cited first, 19 of 19)

- [Practical Kernel Tests of Conditional Independence](https://mlsanity.com/paper/2402.13196/) (2024)
- [A Practical Introduction to Kernel Discrepancies: MMD, HSIC & KSD](https://mlsanity.com/paper/2503.04820/) (2025)
- [Doubly Robust Conditional Independence Testing with Generative Neural Networks](https://mlsanity.com/paper/2407.17694/) (2024)
- [On the Hardness of Conditional Independence Testing In Practice](https://mlsanity.com/paper/2512.14000/) (2025)
- [The Conditional Cauchy-Schwarz Divergence with Applications to Time-Series Data and Sequential Decision Making](https://mlsanity.com/paper/2301.08970/) (2023)
- [Scalable and Flexible Causal Discovery with an Efficient Test for Adjacency](https://mlsanity.com/paper/2406.09177/) (2024)
- [Conditionally Invariant Representation Learning for Disentangling Cellular Heterogeneity](https://mlsanity.com/paper/2307.00558/) (2023)
- [Deep Nonparametric Conditional Independence Tests for Images](https://mlsanity.com/paper/2411.06140/) (2024)
- [AI Alignment in Medical Imaging: Unveiling Hidden Biases Through Counterfactual Analysis](https://mlsanity.com/paper/2504.19621/) (2025)
- [Out-of-Distribution Optimality of Invariant Risk Minimization](https://mlsanity.com/paper/2307.11972/) (2023)
- [Learning Conditional Invariances through Non-Commutativity](https://mlsanity.com/paper/2402.11682/) (2024)
- [Toward Scalable and Valid Conditional Independence Testing with Spectral Representations](https://mlsanity.com/paper/2512.19510/) (2025)
- [Structure-agnostic Causal Representation Learning](https://mlsanity.com/paper/2610.00968/) (2026)
- [Returning The Favour: When Regression Benefits From Probabilistic Causal Knowledge](https://mlsanity.com/paper/2301.11214/) (2023)
- [You Only Debias Once: Towards Flexible Accuracy-Fairness Trade-offs at Inference Time](https://mlsanity.com/paper/2503.07066/) (2025)
- [DISCO: Mitigating Bias in Deep Learning with Conditional Distance Correlation](https://mlsanity.com/paper/2506.11653/) (2025)
- [Mixture Proportion Estimation and Weakly-supervised Kernel Test for Conditional Independence](https://mlsanity.com/paper/2604.07191/) (2026)
- [Controlling for Omitted Variable Bias in Deep Neural Networks](https://mlsanity.com/paper/2608.25930/) (2026)
- [Measurement-Gated Provenance Attenuation for Frozen EEG Representations](https://mlsanity.com/paper/2609.32889/) (2026)

## Abstract (arXiv)

We introduce the Conditional Independence Regression CovariancE (CIRCE), a measure of conditional independence for multivariate continuous-valued variables. CIRCE applies as a regularizer in settings where we wish to learn neural features varphi(X) of data X to estimate a target Y, while being conditionally independent of a distractor Z given Y. Both Z and Y are assumed to be continuous-valued but relatively low dimensional, whereas X and its features may be complex and high dimensional. Relevant settings include domain-invariant learning, fairness, and causal learning. The procedure requires just a single ridge regression from Y to kernelized features of Z, which can be done in advance. It is then only necessary to enforce independence of varphi(X) from residuals of this regression, which is possible with attractive estimation properties and consistency guarantees. By contrast, earlier measures of conditional feature dependence require multiple regressions for each step of feature learning, resulting in more severe bias and variance, and greater computational cost. When sufficiently rich features are used, we establish that CIRCE is zero if and only if varphi(X) perp !!! perp Z mid Y. In experiments, we show superior performance to previous methods on challenging benchmarks, including learning conditionally invariant image features.