< Terug naar vorige pagina

Publicatie

Sample-Efficient Model-Free Reinforcement Learning with Off-Policy Critics

Boekbijdrage - Boekhoofdstuk Conferentiebijdrage

Value-based reinforcement-learning algorithms provide state-of-the-art results in model-free discrete-action settings, and tend to outperform actor-critic algorithms. We argue that actor-critic algorithms are limited by their need for an on-policy critic. We propose Bootstrapped Dual Policy Iteration (BDPI), a novel model-free reinforcement-learning algorithm for continuous states and discrete actions, with an actor and several off-policy critics. Off-policy critics are compatible with experience replay, ensuring high sample-efficiency, without the need for off-policy corrections. The actor, by slowly imitating the average greedy policy of the critics, leads to high-quality and state-specific exploration, which we compare to Thompson sampling. Because the actor and critics are fully decoupled, BDPI is remarkably stable, and unusually robust to its hyper-parameters. BDPI is significantly more sample-efficient than Bootstrapped DQN, PPO, and ACKTR, on discrete, continuous and pixel-based tasks.
Boek: Lecture Notes in Artificial Intelligence
Series: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume: 11908
Pagina's: 19-34
Aantal pagina's: 16
ISBN:978-3-030-46132-4
Jaar van publicatie:2020
BOF-keylabel:ja
IOF-keylabel:ja
Toegankelijkheid:Open