mlsanityData to 7 Oct 2026

Sequence to Sequence Learning with Neural Networks

arXiv 1409.3215 ↗First posted 10 Sep 2014cs.CL cs.LGcited by 2,974

Not on the map. Why?

Cited by 42 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Deep Neural Networks (DNNs) are powerful models that have achieved excellent performance on difficult learning tasks. Although DNNs work well whenever large labeled training sets are available, they cannot be used to map sequences to sequences. In this paper, we present a general end-to-end approach to sequence learning that makes minimal assumptions on the sequence structure. Our method uses a multilayered Long Short-Term Memory (LSTM) to map the input sequence to a vector of a fixed dimensionality, and then another deep LSTM to decode the target sequence from the vector. Our main result is that on an English to French translation task from the WMT'14 dataset, the translations produced by the LSTM achieve a BLEU score of 34.8 on the entire test set, where the LSTM's BLEU score was penalized on out-of-vocabulary words. Additionally, the LSTM did not have difficulty on long sentences. For comparison, a phrase-based SMT system achieves a BLEU score of 33.3 on the same dataset. When we used the LSTM to rerank the 1000 hypotheses produced by the aforementioned SMT system, its BLEU score increases to 36.5, which is close to the previous best result on this task. The LSTM also learned sensible phrase and sentence representations that are sensitive to word order and are relatively invariant to the active and the passive voice. Finally, we found that reversing the order of the words in all source sentences (but not target sentences) improved the LSTM's performance markedly, because doing so introduced many short term dependencies between the source and the target sentence which made the optimization problem easier.

Key facts. Cited by 2,974 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Sep 2014.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2017: 1; 2018: 6; 2019: 2; 2020: 4; 2021: 812; 2022: 597; 2023: 572; 2024: 466; 2025: 358; 2026: 156.

2,974 papers.

Sort:

By area:

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  7. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  8. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  9. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  10. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  11. 2021Variational Diffusion Modelscited by 1,145
  12. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
38 more
  1. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  2. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  3. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  4. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  5. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  6. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  7. 2021Editing Factual Knowledge in Language Modelscited by 548
  8. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  9. 2024Large Language Models: A Surveycited by 506
  10. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  11. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  12. 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 465
  13. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 446
  14. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  15. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
  16. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  17. 2021A Survey of Transformerscited by 399
  18. 2021MOTR: End-to-End Multiple-Object Tracking with Transformercited by 379
  19. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 376
  20. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 362
  21. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  22. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  23. 2021Choose a Transformer: Fourier or Galerkincited by 333
  24. 2021Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecastingcited by 332
  25. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  26. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 319
  27. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  28. 2021Spectral Temporal Graph Neural Network for Multivariate Time-series Forecastingcited by 298
  29. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  30. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  31. 2022Draft, Sketch, and Prove: Guiding Formal Theorem Provers with Informal Proofscited by 278
  32. 2022Transformer for Partial Differential Equations' Operator Learningcited by 270
  33. 2022Transformer Memory as a Differentiable Search Indexcited by 242
  34. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  35. 2023Neural General Circulation Models for Weather and Climatecited by 232
  36. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  37. 2024Generative Representational Instruction Tuningcited by 227
  38. 2022A Contrastive Framework for Neural Text Generationcited by 224

50 most cited of 2,974 All 2,974 (JSON)