# Limits of End-to-End Learning

arXiv: https://arxiv.org/abs/1704.08305 · HTML: https://mlsanity.com/paper/1704.08305/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Apr 2017.

## Built on by (most cited first, 30 of 35)

- [The pitfalls of next-token prediction](https://mlsanity.com/paper/2403.06963/) (2024)
- [Sub-Task Decomposition Enables Learning in Sequence to Sequence Tasks](https://mlsanity.com/paper/2204.02892/) (2022)
- [A posteriori learning for quasi-geostrophic turbulence parametrization](https://mlsanity.com/paper/2204.03911/) (2022)
- [Predicting the Future of AI with AI: High-quality link prediction in an exponentially growing knowledge network](https://mlsanity.com/paper/2210.00881/) (2022)
- [RefSum: Refactoring Neural Summarization](https://mlsanity.com/paper/2104.07210/) (2021)
- [Differentiable Spatial Planning using Transformers](https://mlsanity.com/paper/2112.01010/) (2021)
- [Revisiting the Boundary between ASR and NLU in the Age of Conversational Dialog Systems](https://mlsanity.com/paper/2112.05842/) (2021)
- [Automated Evolutionary Approach for the Design of Composite Machine Learning Pipelines](https://mlsanity.com/paper/2106.15397/) (2021)
- [Trust in AI: Interpretability is not necessary or sufficient, while black-box interaction is necessary and sufficient](https://mlsanity.com/paper/2202.05302/) (2022)
- [Domain Generalization in Machine Learning Models for Wireless Communications: Concepts, State-of-the-Art, and Open Issues](https://mlsanity.com/paper/2303.08106/) (2023)
- [Human-inspired Perspectives: A Survey on AI Long-term Memory](https://mlsanity.com/paper/2411.00489/) (2024)
- [Soft Prompt Decoding for Multilingual Dense Retrieval](https://mlsanity.com/paper/2305.09025/) (2023)
- [OneLog: Towards End-to-End Training in Software Log Anomaly Detection](https://mlsanity.com/paper/2104.07324/) (2021)
- [Embracing the black box: Heading towards foundation models for causal discovery from time series data](https://mlsanity.com/paper/2402.09305/) (2024)
- [Streaming Machine Learning and Online Active Learning for Automated Visual Inspection](https://mlsanity.com/paper/2110.09396/) (2021)
- [Establishing Trustworthiness: Rethinking Tasks and Model Evaluation](https://mlsanity.com/paper/2310.05442/) (2023)
- [Active Learning for Automated Visual Inspection of Manufactured Products](https://mlsanity.com/paper/2109.02469/) (2021)
- [End-to-End Analysis of Charge Stability Diagrams with Transformers](https://mlsanity.com/paper/2508.15710/) (2025)
- [Enhanced Behavioral Cloning Based self-driving Car Using Transfer Learning](https://mlsanity.com/paper/2007.05740/) (2020)
- [Exploratory State Representation Learning](https://mlsanity.com/paper/2109.13596/) (2021)
- [AI Thinking: A framework for rethinking artificial intelligence in practice](https://mlsanity.com/paper/2409.12922/) (2024)
- [Machine Learning Applications on Neuroimaging for Diagnosis and Prognosis of Epilepsy: A Review](https://mlsanity.com/paper/2102.03336/) (2021)
- [Attention-Based Recommendation On Graphs](https://mlsanity.com/paper/2201.05499/) (2022)
- [A Learning Framework for Bandwidth-Efficient Distributed Inference in Wireless IoT](https://mlsanity.com/paper/2203.09631/) (2022)
- [Classification of Phonological Parameters in Sign Languages](https://mlsanity.com/paper/2205.12072/) (2022)
- [Robust Anomaly Map Assisted Multiple Defect Detection with Supervised Classification Techniques](https://mlsanity.com/paper/2212.09352/) (2022)
- [Adaptive SpikeDeep-Classifier: Self-organizing and self-supervised machine learning algorithm for online spike sorting](https://mlsanity.com/paper/2304.01355/) (2023)
- [CycleSL: Server-Client Cyclical Update Driven Scalable Split Learning](https://mlsanity.com/paper/2511.18611/) (2025)
- [A Brief Review of Machine Learning Techniques for Protein Phosphorylation Sites Prediction](https://mlsanity.com/paper/2108.04951/) (2021)
- [InFi: End-to-End Learning to Filter Input for Resource-Efficiency in Mobile-Centric Inference](https://mlsanity.com/paper/2209.13873/) (2022)

## Abstract (arXiv)

End-to-end learning refers to training a possibly complex learning system by applying gradient-based learning to the system as a whole. End-to-end learning system is specifically designed so that all modules are differentiable. In effect, not only a central learning machine, but also all "peripheral" modules like representation learning and memory formation are covered by a holistic learning process. The power of end-to-end learning has been demonstrated on many tasks, like playing a whole array of Atari video games with a single architecture. While pushing for solutions to more challenging tasks, network architectures keep growing more and more complex. In this paper we ask the question whether and to what extent end-to-end learning is a future-proof technique in the sense of scaling to complex and diverse data processing architectures. We point out potential inefficiencies, and we argue in particular that end-to-end learning does not make optimal use of the modular design of present neural networks. Our surprisingly simple experiments demonstrate these inefficiencies, up to the complete breakdown of learning.