Policy Gradient Methods for Reinforcement Learning

Considering Actor-Critic for handling the decision making of agents, my colleague and I started an interesting discussion about policy gradient methods in reinforcement learning. To clarify the concepts of our discussion, I summarised the lesson I learned in this short post.

Consider The Policy Parameter Space.

Actor-Critic (3).jpg
yellow dots – possible policies, orange dots – closer policies, black dot – initial policy or delta theta, grey dots – evaluated policies, red dot – optimal policy grey arrows – best direction, grey dotted arrows – possible directions. 

The general idea of policy gradient methods is to compute the gradient with respect to the parameters of the policy. However, in order to do that, we need to know the reward function and the transition function of the policy. Using a basic approach, we try different directions by randomly guessing policies and further evaluating the policies that we get.

If we sample enough directions, we will eventually get to the best direction (or close enough to it). 

The learning rate helps us to test closer policies instead of moving completely towards a new policy. Why do policy gradient is not so easy in RL? The advantage of the policy gradient is that we don’t need exploration as in classic RL problems. The disadvantages are that (1) we need to compute many directions to approximate the gradient and (2) we are never certain that we have found the best direction for updating our policies.

Actor-Critic algorithms try to address this problem by limiting the number of policies to evaluate;

  • The “Critic” part of the algorithm, that could be the action-value (the Q value) or the state-value (the V value), approximate the quality of the new policies.
  • The “Actor” updates the policy distribution in the direction suggested by the “Critic”.

Regarding Actor-Critic algorithms, I found the following two posts interesting and easy to understand:

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s