Learning with Opponent-Learning Awareness (LOLA), introduced by Foerster et al. in 2018, has each agent take into account the anticipated learning of the other agents, rather than treating them as a fixed part of the environment.
“each agent shapes the anticipated learning of the other agents in the environment”
Foerster et al., Learning with Opponent-Learning Awareness, 2018
The headline result is that when two LOLA agents meet in the iterated prisoner’s dilemma they arrive at tit-for-tat, so cooperation emerges rather than being designed in. The authors also show the update rule can be estimated with an extension of the likelihood-ratio policy gradient, which keeps the method usable for model-free reinforcement learning and lets it scale to larger parameter and input spaces, and they apply it to a grid world with an embedded social dilemma.
What interests me in it is the framing. An agent that models how the other agent learns is doing something closer to teaching than to competing, which puts the method unexpectedly near work on pedagogical agents.
