mlsanityData to 7 Oct 2026

HarmonyDream: Task Harmonization Inside World Models

arXiv 2310.00344 ↗First posted 30 Sep 2023cs.LGcited by 22

On the map: World models → Latent dynamics models and model-based RL

also in 1 more
Abstract (arXiv)

Model-based reinforcement learning (MBRL) holds the promise of sample-efficient learning by utilizing a world model, which models how the environment works and typically encompasses components for two tasks: observation modeling and reward modeling. In this paper, through a dedicated empirical investigation, we gain a deeper understanding of the role each task plays in world models and uncover the overlooked potential of sample-efficient MBRL by mitigating the domination of either observation or reward modeling. Our key insight is that while prevalent approaches of explicit MBRL attempt to restore abundant details of the environment via observation models, it is difficult due to the environment's complexity and limited model capacity. On the other hand, reward models, while dominating implicit MBRL and adept at learning compact task-centric dynamics, are inadequate for sample-efficient learning without richer learning signals. Motivated by these insights and discoveries, we propose a simple yet effective approach, HarmonyDream, which automatically adjusts loss coefficients to maintain task harmonization, i.e. a dynamic equilibrium between the two tasks in world model learning. Our experiments show that the base MBRL method equipped with HarmonyDream gains 10%-69% absolute performance boosts on visual robotic tasks and sets a new state-of-the-art result on the Atari 100K benchmark. Code is available at https://github.com/thuml/HarmonyDream.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 53 of its 60 references that are in the map. First posted on arXiv 30 Sep 2023.

Builds on

53 of 60 references.

Closest ancestors · 14

Same direction, newest first

  1. 2023Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learning29 May
  2. 2023Transformer-based World Models Are Happy With 100k Interactions13 Mar
  3. 2023Mastering Diverse Domains through World Models10 Jan
  4. 2022Transformers are Sample-Efficient World Models1 Sep
  5. 2022Denoised MDPs: Learning World Models Better Than the World Itself30 Jun
  6. 2022DayDreamer: World Models for Physical Robot Learning28 Jun
  7. 2021Procedural Generalization by Planning with Self-Supervised World Models2 Nov
  8. 2021Mastering Atari Games with Limited Data30 Oct
  9. 2021DreamerPro: Reconstruction-Free Model-Based Reinforcement Learning with Prototypical Representations27 Oct
  10. 2020Mastering Atari with Discrete World Models5 Oct
  11. 2020Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstruction29 Jul
  12. 2019Dream to Control: Learning Behaviors by Latent Imagination3 Dec
2 more
  1. 2018Recurrent World Models Facilitate Policy Evolution4 Sep
  2. 2017Value Prediction Network11 Jul

Other references · 39

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  3. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  5. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  6. 2016Progressive Neural Networkscited by 1,332
  7. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  8. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  9. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  10. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  11. 2017Mixed Precision Trainingcited by 1,135
  12. 2018DeepMind Control Suitecited by 935
27 more
  1. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  2. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 791
  3. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  4. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  5. 2018End-to-End Multi-Task Learning with Attentioncited by 533
  6. 2019Model-Based Reinforcement Learning for Ataricited by 495
  7. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  8. 2016Cross-stitch Networks for Multi-task Learningcited by 466
  9. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 441
  10. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  11. 2022Temporal Difference Learning for Model Predictive Controlcited by 375
  12. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  13. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  14. 2017Distral: Robust Multitask Reinforcement Learningcited by 231
  15. 2022Multi-Task Learning as a Bargaining Gamecited by 209
  16. 2021Benchmarking the Spectrum of Agent Capabilitiescited by 193
  17. 2022Masked World Models for Visual Controlcited by 169
  18. 2020Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Modelscited by 167
  19. 2021Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisationcited by 120
  20. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  21. 2020The Value Equivalence Principle for Model-Based Reinforcement Learningcited by 88
  22. 2020Objective Mismatch in Model-based Reinforcement Learningcited by 87
  23. 2021Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulationcited by 85
  24. 2021Temporal Predictive Coding For Model-Based Planning In Latent Spacecited by 54
  25. 2018Natural Environment Benchmarks for Reinforcement Learningcited by 32
  26. 2020Measuring Visual Generalization in Continuous Control from Pixelscited by 20
  27. 2023Learning from Visual Observation via Offline Pretrained State-to-Go Transformercited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 10; 2026: 8.

22 papers.

Sort:

By area:

  1. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 136
  2. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  3. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 50
  4. 2026World Action Models: The Next Frontier in Embodied AIcited by 46
  5. 2026VideoWorld 2: Learning Transferable Knowledge from Real-world Videoscited by 11
  6. 2025Four Principles for Physically Interpretable World Modelscited by 10
  7. 2025Discrete Codebook World Models for Continuous Controlcited by 9
  8. 2025FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Makingcited by 8
  9. 2024CompilerDream: Learning a Compiler World Model for General Code Optimizationcited by 6
  10. 2025Disentangled World Models: Learning to Transfer Semantic Knowledge from Distracting Videos for Reinforcement Learningcited by 6
  11. 2025SAMPO:Scale-wise Autoregression with Motion PrOmpt for generative world modelscited by 4
  12. 2024Representing Positional Information in Generative World Models for Object Manipulationcited by 3
10 more
  1. 2025EDELINE: Enhancing Memory in Diffusion-based World Models via Linear-Time Sequence Modelingcited by 3
  2. 2025One Model for All Tasks: Leveraging Efficient World Models in Multi-Task Planningcited by 3
  3. 2026OpenWorldLib: A Unified Codebase and Definition of Advanced World Modelscited by 3
  4. 2026Morphology-Conditioned World Model for Cross-Embodiment Quadrupedal Locomotioncited by 3
  5. 2025Efficient Reinforcement Learning Through Adaptively Pretrained Visual Encodercited by 2
  6. 2025DyMoDreamer: World Modeling with Dynamic Modulationcited by 2
  7. 2026Mixture-of-World Models: Scaling Multi-Task Reinforcement Learning with Modular Latent Dynamicscited by 1
  8. 2026Pessimistic Auxiliary Policy for Offline Reinforcement Learningcited by 0
  9. 2026Probing the Impact of Scale on Data-Efficient, Generalist Transformer World Models for Ataricited by 0
  10. 2026Transferable Delay-Aware Reinforcement Learning via Implicit Causal Graph Modelingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022On the Feasibility of Cross-Task Transfer with Model-Based Reinforcement Learning
  2. 2024Task-Aware Harmony Multi-Task Decision Transformer for Offline Reinforcement Learning
  3. 2025RoboScape-R: Unified Reward-Observation World Models for Generalizable Robotics Training via RL
  4. 2024LIMT: Language-Informed Multi-Task Visual World Models
  5. 2025Accelerating Model-Based Reinforcement Learning with State-Space World Models
  6. 2026Scaling Tasks, Not Samples: Mastering Humanoid Control through Multi-Task Model-Based Reinforcement Learning
  7. 2026Q-Learning With World Models
  8. 2024PWM: Policy Learning with Multi-Task World Models