mlsanityData to 14 Oct 2026

Using Sequences of Life-events to Predict Human Lives

arXiv 2306.03009 ↗First posted 5 Jun 2023stat.ML cs.LG stat.APcited by 27

Not on the map. Why?

Abstract (arXiv)

Over the past decade, machine learning has revolutionized computers' ability to analyze text through flexible computational models. Due to their structural similarity to written language, transformer-based architectures have also shown promise as tools to make sense of a range of multi-variate sequences from protein-structures, music, electronic health records to weather-forecasts. We can also represent human lives in a way that shares this structural similarity to language. From one perspective, lives are simply sequences of events: People are born, visit the pediatrician, start school, move to a new location, get married, and so on. Here, we exploit this similarity to adapt innovations from natural language processing to examine the evolution and predictability of human lives based on detailed event sequences. We do this by drawing on arguably the most comprehensive registry data in existence, available for an entire nation of more than six million individuals across decades. Our data include information about life-events related to health, education, occupation, income, address, and working hours, recorded with day-to-day resolution. We create embeddings of life-events in a single vector space showing that this embedding space is robust and highly structured. Our models allow us to predict diverse outcomes ranging from early mortality to personality nuances, outperforming state-of-the-art models by a wide margin. Using methods for interpreting deep learning models, we probe the algorithm to understand the factors that enable our predictions. Our framework allows researchers to identify new potential mechanisms that impact life outcomes and associated possibilities for personalized interventions.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Builds on 46 of its 126 references that are in the map. First posted on arXiv 5 Jun 2023.

Builds on

46 of 126 references.

References · 46

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  9. 2020Longformer: The Long-Document Transformercited by 3,413
  10. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  11. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
  12. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
34 more
  1. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  2. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,917
  3. 2020Rethinking Attention with Performerscited by 1,601
  4. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,319
  5. 2019When Does Label Smoothing Help?cited by 988
  6. 2017Selective Classification for Deep Neural Networkscited by 852
  7. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  8. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  9. 2019How to Fine-Tune BERT for Text Classification?cited by 411
  10. 2014Deep Speech: Scaling up end-to-end speech recognitioncited by 354
  11. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 264
  12. 2019BEHRT: Transformer for Electronic Health Recordscited by 263
  13. 2020A Survey on the Explainability of Supervised Machine Learningcited by 227
  14. 2019Time2Vec: Learning a Vector Representation of Timecited by 213
  15. 2017All-but-the-Top: Simple and Effective Postprocessing for Word Representationscited by 212
  16. 2018Music Transformercited by 211
  17. 2020Understanding How Dimension Reduction Tools Work: An Empirical Approach to Deciphering t-SNE, UMAP, TriMAP, and PaCMAP for Data Visualizationcited by 173
  18. 2020A Diagnostic Study of Explainability Techniques for Text Classificationcited by 159
  19. 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
  20. 2020The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?cited by 122
  21. 2022Should You Mask 15% in Masked Language Modeling?cited by 118
  22. 2020End-to-end Lane Shape Prediction with Transformerscited by 111
  23. 2018The Geometry of Culture: Analyzing Meaning through Word Embeddingscited by 87
  24. 2018From Word to Sense Embeddings: A Survey on Vector Representations of Meaningcited by 49
  25. 2020On Interpretability of Deep Learning based Skin Lesion Classifiers using Concept Activation Vectorscited by 39
  26. 2020Multi-Loss Weighting with Coefficient of Variationscited by 36
  27. 2019Saliency-driven Word Alignment Interpretation for Neural Machine Translationcited by 27
  28. 2021Learning to Remove: Towards Isotropic Pre-trained BERT Embeddingcited by 23
  29. 2018Sigsoftmax: Reanalysis of the Softmax Bottleneckcited by 20
  30. 2021MELONS: generating melody with long-term structure using transformers and structure graphcited by 18
  31. 2022Class Distance Weighted Cross-Entropy Loss for Ulcerative Colitis Severity Estimationcited by 17
  32. 2021Towards physically consistent data-driven weather forecasting: Integrating data assimilation with equivariance-preserving deep spatial transformerscited by 15
  33. 2018Beyond Weight Tying: Learning Joint Input-Output Embeddings for Neural Machine Translationcited by 5
  34. 2020Improving Positive Unlabeled Learning: Practical AUL Estimation and New Training Method for Extremely Imbalanced Data Setscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 14; 2026: 5.

27 papers.

Sort:

  1. 2024Zero Shot Health Trajectory Prediction Using Transformercited by 77
  2. 2025The Benchmarking Epistemology: Validity Theory for Evaluating Machine Learning Modelscited by 13
  3. 2024A Population-to-individual Tuning Framework for Adapting Pretrained LM to On-device User Intent Predictioncited by 6
  4. 2024Are KANs Effective for Multivariate Time Series Forecasting?cited by 6
  5. 2024Empathy and the Right to Be an Exception: What LLMs Can and Cannot Docited by 4
  6. 2024Estimating Wage Disparities Using Foundation Modelscited by 4
  7. 2024Combining the Strengths of Dutch Survey and Register Data in a Data Challenge to Predict Fertility (PreFer)cited by 3
  8. 2025Addressing Longstanding Challenges in Cognitive Science with Language Modelscited by 3
  9. 2025Tuning Language Models for Robust Prediction of Diverse User Behaviorscited by 2
  10. 2024Social Media Use is Predictable from App Sequences: Using LSTM and Transformer Neural Networks to Model Habitual Behaviorcited by 1
  11. 2025Improving Representation Learning of Complex Critical Care Data with ICU-BERTcited by 1
  12. 2025Explainable embeddings with Distance Explainercited by 1
15 more
  1. 2025Why can't Epidemiology be automated (yet)?cited by 1
  2. 2025Using Text-Based Life Trajectories from Swedish Register Data to Predict Residential Mobility with Pretrained Transformerscited by 1
  3. 2026How Hyper-Datafication Impacts the Sustainability Costs in Frontier AIcited by 1
  4. 2024On the Unknowable Limits to Predictioncited by 0
  5. 2025Large language model as user daily behavior data generator: balancing population diversity and individual personalitycited by 0
  6. 2025Scaling Electronic Health Record Foundation Models for Population Health Managementcited by 0
  7. 2025The Book of Life approach: Enabling richness and scale for life course researchcited by 0
  8. 2025Population-Scale Network Embeddings Expose Educational Divides in Network Structure Related to Right-Wing Populist Votingcited by 0
  9. 2025ASCENDgpt: A Phenotype-Aware Transformer Model for Cardiovascular Risk Prediction from Electronic Health Recordscited by 0
  10. 2025The Intercepted Self: How Generative AI Challenges the Dynamics of the Relational Selfcited by 0
  11. 2025Deep Generative Model for Human Mobility Behaviorcited by 0
  12. 2026LLMs Reading the Rhythms of Daily Life: Aligned Understanding for Behavior Prediction and Generationcited by 0
  13. 2026SAGA: A Sequence-Adaptive Generative Architecture for Multi-Horizon Probabilistic Forecasting with Adaptive Temporal Conformal Predictioncited by 0
  14. 2026LifeSentence: Language models can encode human life course trajectories from longitudinal panel datacited by 0
  15. 2026TEDDY: A Pediatric Foundation Model for Risk Forewarning from ICD-Coded Diagnostic Historiescited by 0