A compelling way to reduce the cost of real experience in reinforcement learning is to learn policies on data synthesised by models. The authors of Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search propose CF-GPS, an algorithm for learning policies in POMDPs from off-policy experience.
The novelty lies in considering alternative outcomes, which lets the algorithm make better use of the experience it already has. Their empirical evaluation reports improvements in both policy evaluation and policy search on a non-trivial grid-world task.
Buesing et al. introduce the problem through an everyday example: re-evaluating a job offer after having made a regrettable decision.
The appeal is that the method pays for better policies with reasoning rather than with more interaction, which matters in any setting where collecting experience is the expensive part.
