# Cross-attention Secretly Performs Orthogonal Alignment in Recommendation Models

arXiv: https://arxiv.org/abs/2510.09435 · HTML: https://mlsanity.com/paper/2510.09435/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 36 references that are in the map.
- First posted on arXiv 10 Oct 2025.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [Transformer Interpretability Beyond Attention Visualization](https://mlsanity.com/paper/2012.09838/) (2020)
- [DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference](https://mlsanity.com/paper/2004.12993/) (2020)
- [DeepViT: Towards Deeper Vision Transformer](https://mlsanity.com/paper/2103.11886/) (2021)
- [What the DAAM: Interpreting Stable Diffusion Using Cross Attention](https://mlsanity.com/paper/2210.04885/) (2022)
- [Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoders](https://mlsanity.com/paper/2403.03952/) (2024)
- [What to align in multimodal contrastive learning?](https://mlsanity.com/paper/2409.07402/) (2024)
- [Cross-Modal Retrieval: A Systematic Review of Methods and Future Directions](https://mlsanity.com/paper/2308.14263/) (2023)
- [Harnessing Multimodal Large Language Models for Multimodal Sequential Recommendation](https://mlsanity.com/paper/2408.09698/) (2024)
- [MiNet: Mixed Interest Network for Cross-Domain Click-Through Rate Prediction](https://mlsanity.com/paper/2008.02974/) (2020)
- [Voila-A: Aligning Vision-Language Models with User's Gaze Attention](https://mlsanity.com/paper/2401.09454/) (2023)
- [Diffusion Model with Cross Attention as an Inductive Bias for Disentanglement](https://mlsanity.com/paper/2402.09712/) (2024)
- [It's Not a Modality Gap: Characterizing and Addressing the Contrastive Gap](https://mlsanity.com/paper/2405.18570/) (2024)
- [Triple Sequence Learning for Cross-domain Recommendation](https://mlsanity.com/paper/2304.05027/) (2023)
- [Image Fusion for Cross-Domain Sequential Recommendation](https://mlsanity.com/paper/2502.15694/) (2024)
- [RecGURU: Adversarial Learning of Generalized User Representations for Cross-Domain Recommendation](https://mlsanity.com/paper/2111.10093/) (2021)
- [A Survey on Cross-Domain Sequential Recommendation](https://mlsanity.com/paper/2401.04971/) (2024)
- [Mixed Attention Network for Cross-domain Sequential Recommendation](https://mlsanity.com/paper/2311.08272/) (2023)
- [Bridge the Domains: Large Language Models Enhanced Cross-domain Sequential Recommendation](https://mlsanity.com/paper/2504.18383/) (2025)
- [Hiformer: Heterogeneous Feature Interactions Learning with Transformers for Recommender Systems](https://mlsanity.com/paper/2311.05884/) (2023)
- [One for All, All for One: Learning and Transferring User Embeddings for Cross-Domain Recommendation](https://mlsanity.com/paper/2211.11964/) (2022)
- [A Collaborative Transfer Learning Framework for Cross-domain Recommendation](https://mlsanity.com/paper/2306.16425/) (2023)
- [Cracking the Code of Negative Transfer: A Cooperative Game Theoretic Approach for Cross-Domain Sequential Recommendation](https://mlsanity.com/paper/2311.13188/) (2023)
- [ABXI: Invariant Interest Adaptation for Task-Guided Cross-Domain Sequential Recommendation](https://mlsanity.com/paper/2501.15118/) (2025)
- [Formalizing Multimedia Recommendation through Multimodal Deep Learning](https://mlsanity.com/paper/2309.05273/) (2023)

## Abstract (arXiv)

Cross-domain sequential recommendation (CDSR) aims to align heterogeneous user behavior sequences collected from different domains. While cross-attention is widely used to enhance alignment and improve recommendation performance, its underlying mechanism is not fully understood. Most researchers interpret cross-attention as residual alignment, where the output is generated by removing redundant and preserving non-redundant information from the query input by referencing another domain data which is input key and value. Beyond the prevailing view, we introduce Orthogonal Alignment, a phenomenon in which cross-attention discovers novel information that is not present in the query input, and further argue that those two contrasting alignment mechanisms can co-exist in recommendation models We find that when the query input and output of cross-attention are orthogonal, model performance improves over 300 experiments. Notably, Orthogonal Alignment emerges naturally, without any explicit orthogonality constraints. Our key insight is that Orthogonal Alignment emerges naturally because it improves scaling law. We show that baselines additionally incorporating cross-attention module outperform parameter-matched baselines, achieving a superior accuracy-per-model parameter. We hope these findings offer new directions for parameter-efficient scaling in multi-modal research.