# Exploring new ways: Enforcing representational dissimilarity to learn new features and reduce error consistency

arXiv: https://arxiv.org/abs/2307.02516 · HTML: https://mlsanity.com/paper/2307.02516/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 26 references that are in the map.
- First posted on arXiv 5 Jul 2023.

## Builds on (most cited first, 15 of 15)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Git Re-Basin: Merging Models modulo Permutation Symmetries](https://mlsanity.com/paper/2209.04836/) (2022)
- [Insights on representational similarity in neural networks with canonical correlation](https://mlsanity.com/paper/1806.05759/) (2018)
- [Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth](https://mlsanity.com/paper/2010.15327/) (2020)
- [Understanding image representations by measuring their equivariance and equivalence](https://mlsanity.com/paper/1411.5908/) (2014)
- [Convergent Learning: Do different neural networks learn the same representations?](https://mlsanity.com/paper/1511.07543/) (2015)
- [Improving Adversarial Robustness via Promoting Ensemble Diversity](https://mlsanity.com/paper/1901.08846/) (2019)
- [Revisiting Model Stitching to Compare Neural Representations](https://mlsanity.com/paper/2106.07682/) (2021)
- [Relative representations enable zero-shot latent space communication](https://mlsanity.com/paper/2209.15430/) (2022)
- [Similarity of Neural Network Models: A Survey of Functional and Representational Measures](https://mlsanity.com/paper/2305.06329/) (2023)
- [Similarity and Matching of Neural Network Representations](https://mlsanity.com/paper/2110.14633/) (2021)
- [Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representation](https://mlsanity.com/paper/1810.11750/) (2018)
- [No One Representation to Rule Them All: Overlapping Features of Training Methods](https://mlsanity.com/paper/2110.12899/) (2021)
- [Automatic Shortcut Removal for Self-Supervised Representation Learning](https://mlsanity.com/paper/2002.08822/) (2020)

## Abstract (arXiv)

Independently trained machine learning models tend to learn similar features. Given an ensemble of independently trained models, this results in correlated predictions and common failure modes. Previous attempts focusing on decorrelation of output predictions or logits yielded mixed results, particularly due to their reduction in model accuracy caused by conflicting optimization objectives. In this paper, we propose the novel idea of utilizing methods of the representational similarity field to promote dissimilarity during training instead of measuring similarity of trained models. To this end, we promote intermediate representations to be dissimilar at different depths between architectures, with the goal of learning robust ensembles with disjoint failure modes. We show that highly dissimilar intermediate representations result in less correlated output predictions and slightly lower error consistency, resulting in higher ensemble accuracy. With this, we shine first light on the connection between intermediate representations and their impact on the output predictions.