Easing Optimization Paths: a Circuit Perspective
Not on the map. Why?
Abstract (arXiv)
Gradient descent is the method of choice for training large artificial intelligence systems. As these systems become larger, a better understanding of the mechanisms behind gradient training would allow us to alleviate compute costs and help steer these systems away from harmful behaviors. To that end, we suggest utilizing the circuit perspective brought forward by mechanistic interpretability. After laying out our intuition, we illustrate how it enables us to design a curriculum for efficient learning in a controlled setting. The code is available at urlhttps://github.com/facebookresearch/pal.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 10 of its 26 references that are in the map. First posted on arXiv 4 Jan 2025.
Builds on
10 of 26 references.
References · 10
- 2017Attention Is All You Needcited by 59,932
- 2019Cross-lingual Language Model Pretrainingcited by 1,088
- 2019Competence-based Curriculum Learning for Neural Machine Translationcited by 201
- 2024Why Transformers Need Adam: A Hessian Perspectivecited by 123
- 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
- 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 91
- 2017How hard can it be? Estimating the difficulty of visual search in an imagecited by 52
- 2024SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attentioncited by 43
- 2023Provable Advantage of Curriculum Learning on Parity Targets with Mixed Inputscited by 22
- 2023Leveraging Ensemble Diversity for Robust Self-Training in the Presence of Sample Selection Biascited by 7
Built on by
Not cited yet.