mlsanityData to 7 Oct 2026

Model Alignment Search

arXiv 2501.06164 ↗First posted 10 Jan 2025cs.LG cs.AIcited by 3

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

When can we say that two neural systems perform a task in the same way? What nuances do we miss when we fail to causally probe the representations of the systems, and how do we establish bidirectional causal relationships? In this work, we introduce a method that bidirectionally transfers neural activity between artificial neural networks and uses their resulting behavior as a measure of functional similarity. We first show that the method can be used to transfer the behavior from one frozen Neural Network (NN) to another in a manner similar to model stitching, and we show how the method can differ from correlative similarity measures like Representational Similarity Analysis. Next, we empirically and theoretically show how the method can be equivalent to model stitching when desired, or it can take a form that has a more restrictive focus to shared causal information; in both forms, it reduces the number of required matrices for a comparison of n models to be linear in n. We then present a case study on number-related tasks showing that the method can be used to examine specific subtypes of causal information demonstrating that numbers can be encoded differently in recurrent models depending on the task, and we present another case study showing that MAS can reveal misalignment in fine-tuned DeepSeek-r1-Qwen-1.5B models. Lastly, we augment the loss function with a counterfactual latent (CL) auxiliary objective to improve causal relevance when one of the two networks is causally inaccessible (as is often the case in comparisons with biological networks). We use our results to encourage the use of causal methods in neural similarity analyses and to suggest future explorations of network similarity methodology for model misalignment.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 42 of its 90 references that are in the map. First posted on arXiv 10 Jan 2025.

Builds on

42 of 90 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2016Layer Normalizationcited by 5,579
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  8. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  11. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  12. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
30 more
  1. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  2. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  3. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  4. 2021Causal Abstractions of Neural Networkscited by 401
  5. 2024The Platonic Representation Hypothesiscited by 362
  6. 2014Understanding image representations by measuring their equivariance and equivalencecited by 240
  7. 2023ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversationcited by 224
  8. 2021Revisiting Model Stitching to Compare Neural Representationscited by 203
  9. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 197
  10. 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 194
  11. 2020Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Biascited by 183
  12. 2022Relative representations enable zero-shot latent space communicationcited by 175
  13. 2023Getting aligned on representational alignmentcited by 142
  14. 2023Similarity of Neural Network Models: A Survey of Functional and Representational Measurescited by 128
  15. 2020What shapes feature representations? Exploring datasets, architectures, and trainingcited by 125
  16. 2021Generalized Shape Metrics on Neural Representationscited by 109
  17. 2021Inducing Causal Structure for Interpretable Neural Networkscited by 107
  18. 2023Interpretability at Scale: Identifying Causal Mechanisms in Alpacacited by 102
  19. 2022Reliability of CKA as a Similarity Measure in Deep Learningcited by 82
  20. 2023Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patchingcited by 75
  21. 2024Towards Universality: Studying Mechanistic Similarity Across Language Model Architecturescited by 31
  22. 2024Assessing and Learning Alignment of Unimodal Vision and Language Modelscited by 31
  23. 2023Do Vision and Language Models Share Concepts? A Vector Space Alignment Studycited by 27
  24. 2024Learned feature representations are biased by complexity, learning order, position, and morecited by 22
  25. 2021Explanatory models in neuroscience: Part 1 -- taking mechanistic abstraction seriouslycited by 18
  26. 2023Soft Matching Distance: A metric on neural representations that captures single-neuron tuningcited by 16
  27. 2024Correcting Biased Centered Kernel Alignment Measures in Biological and Artificial Neural Networkscited by 16
  28. 2023Model Stitching: Looking For Functional Similarity Between Representationscited by 10
  29. 2024Differentiable Optimization of Similarity Scores Between Models and Brainscited by 10
  30. 2024Counting in Small Transformers: The Delicate Interplay between Attention and Feed-Forward Layerscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Addressing divergent representations from causal interventions on neural networkscited by 6
  2. 2025Cross-Model Semantics in Representation Learningcited by 2
  3. 2026Portable Semantics, Private Dialects: Reuse and Negative Transfer in Latent Communication Between Language-Model Cellscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024What Makes Two Language Models Think Alike?
  2. 2025Combining Causal Models for More Accurate Abstractions of Neural Networks
  3. 2026Barycentric alignment for instance-level comparison of neural representations
  4. 2026The Triangle of Similarity: A Multi-Faceted Framework for Comparing Neural Network Representations
  5. 2024Dynamical similarity analysis can identify compositional dynamics developing in RNNs
  6. 2026Rashomon Alignment
  7. 2026Symmetries and Causality: Causal Effect Identification Beyond IID Data
  8. 2022Neural State-Space Modeling with Latent Causal-Effect Disentanglement