# Automatic Curriculum Learning With Over-repetition Penalty for Dialogue Policy Learning

arXiv: https://arxiv.org/abs/2012.14072 · HTML: https://mlsanity.com/paper/2012.14072/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Dec 2020.

## Built on by (most cited first, 8 of 8)

- [Curriculum Learning: A Survey](https://mlsanity.com/paper/2101.10382/) (2021)
- [Preview, Attend and Review: Schema-Aware Curriculum Learning for Multi-Domain Dialog State Tracking](https://mlsanity.com/paper/2106.00291/) (2021)
- [A Psychology-based Unified Dynamic Framework for Curriculum Learning](https://mlsanity.com/paper/2408.05326/) (2024)
- [LaMDAgent: An Autonomous Framework for Post-Training Pipeline Optimization via LLM Agents](https://mlsanity.com/paper/2505.21963/) (2025)
- [Investigating Value of Curriculum Reinforcement Learning in Autonomous Driving Under Diverse Road and Weather Conditions](https://mlsanity.com/paper/2103.07903/) (2021)
- [Breaking the Loop: Detecting and Mitigating Denial-of-Service Vulnerabilities in Large Language Models](https://mlsanity.com/paper/2503.00416/) (2025)
- [Scheduled Curiosity-Deep Dyna-Q: Efficient Exploration for Dialog Policy Learning](https://mlsanity.com/paper/2402.00085/) (2024)
- [PMG: Progressive Motion Generation via Sparse Anchor Postures Curriculum Learning](https://mlsanity.com/paper/2504.16722/) (2025)

## Abstract (arXiv)

Dialogue policy learning based on reinforcement learning is difficult to be applied to real users to train dialogue agents from scratch because of the high cost. User simulators, which choose random user goals for the dialogue agent to train on, have been considered as an affordable substitute for real users. However, this random sampling method ignores the law of human learning, making the learned dialogue policy inefficient and unstable. We propose a novel framework, Automatic Curriculum Learning-based Deep Q-Network (ACL-DQN), which replaces the traditional random sampling method with a teacher policy model to realize the dialogue policy for automatic curriculum learning. The teacher model arranges a meaningful ordered curriculum and automatically adjusts it by monitoring the learning progress of the dialogue agent and the over-repetition penalty without any requirement of prior knowledge. The learning progress of the dialogue agent reflects the relationship between the dialogue agent's ability and the sampled goals' difficulty for sample efficiency. The over-repetition penalty guarantees the sampled diversity. Experiments show that the ACL-DQN significantly improves the effectiveness and stability of dialogue tasks with a statistically significant margin. Furthermore, the framework can be further improved by equipping with different curriculum schedules, which demonstrates that the framework has strong generalizability.