An operator view of policy gradient methods

We cast policy gradient methods as the repeated application of two operators: a policy improvement operator \(\mathcal{I}\), which maps any policy \(\pi\) to a better one \(\mathcal{I}\pi\), and a projection operator \(\mathcal{P}\), which finds the best approximation of \(\mathcal{I}\pi\) in the set of realizable policies. We use this framework to introduce operator-based versions of traditional policy gradient methods such as REINFORCE and PPO, which leads to a better understanding of their original counterparts. We also use the understanding we develop of the role of \(\mathcal{I}\) and \(\mathcal{P}\) to propose a new global lower bound of the expected return. This new perspective allows us to further bridge the gap between policy-based and value-based methods, showing how REINFORCE and the Bellman optimality operator, for example, can be seen as two sides of the same coin.