{"id":934452,"date":"2023-04-19T09:08:44","date_gmt":"2023-04-19T16:08:44","guid":{"rendered":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/blog\/unifying-learning-from-preferences-and-demonstration-via-a-ranking-game-for-imitation-learning\/"},"modified":"2023-04-19T09:41:16","modified_gmt":"2023-04-19T16:41:16","slug":"unifying-learning-from-preferences-and-demonstration-via-a-ranking-game-for-imitation-learning","status":"publish","type":"post","link":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/blog\/unifying-learning-from-preferences-and-demonstration-via-a-ranking-game-for-imitation-learning\/","title":{"rendered":"Unifying learning from preferences and demonstration via a ranking game for imitation learning"},"content":{"rendered":"\n
\"Rank<\/figure>\n\n\n\n

For many people, opening door handles or moving a pen between their fingers is a movement that happens multiple times a day, often without much thought. For a robot, however, these movements aren\u2019t always so easy.<\/p>\n\n\n\n

In reinforcement learning, robots learn to perform tasks by exploring their environments, receiving signals along the way that indicate how good their behavior is compared to the desired outcome, or state<\/em>. For the described movements, for example, we can specify a reward function that is +1 when the door is successfully opened or the pen is at the desired orientation and 0 otherwise. But this makes the learning task complicated for the robot since it has to try out various motions before stumbling on the successful outcome, or a reward of +1.<\/p>\n\n\n\n

The imitation learning (IL) paradigm was introduced to mitigate the amount of trial and error. In IL, the robot is provided with demonstrations of a given task performed by an expert from which it can try to learn the task and possibly gain information about the expert\u2019s reward function, or the expert\u2019s intent<\/em>, similar to how people pick up various skills. Yet, learning remains difficult in instances where we only have access to the change enacted by the expert in the world, known as the expert observation<\/em>, and not the precise actions the expert took to achieve the change. Another difficulty the robot faces is that even if it sees infinite expert demonstrations, it can\u2019t fully reason about the intent of the expert\u2014that is, compare whether one of its own learned behaviors is closer to the expert\u2019s than another behavior\u2014as it only knows the best<\/em> behavior and has no notion of ordering over other behaviors.<\/p>\n\n\n\n

<\/div>\n\n\n\n\t
\n\t\t\n\n\t\t

\n\t\tPODCAST SERIES<\/span>\n\t<\/p>\n\t\n\t

\n\t\t\t\t\t\t
\n\t\t\t\t\n\t\t\t\t\t\"Illustrated\n\t\t\t\t<\/a>\n\t\t\t<\/div>\n\t\t\t\n\t\t\t
\n\n\t\t\t\t\t\t\t\t\t

AI Testing and Evaluation: Learnings from Science and Industry<\/h2>\n\t\t\t\t\n\t\t\t\t\t\t\t\t

Discover how Microsoft is learning from other domains to advance evaluation and testing as a pillar of AI governance.<\/p>\n\t\t\t\t\n\t\t\t\t\t\t\t\t

\n\t\t\t\t\t
\n\t\t\t\t\t\t\n\t\t\t\t\t\t\tListen now\t\t\t\t\t\t<\/a>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t<\/div>\n\t<\/div>\n\t\n\n\n

In our paper \u201cA Ranking Game for Imitation Learning<\/a>,\u201d being presented at Transactions on Machine Learning Research 2023 (TMLR (opens in new tab)<\/span><\/a>), we propose a simple and intuitive framework, \\(\\texttt{rank-game}\\), that unifies learning from expert demonstrations and preferences by generalizing a key approach to imitation learning. Giving robots the ability to learn from preferences, obtained by having an expert rank which behavior aligns better with their objectives, allows the learning of more informative reward functions. Our approach, which enabled us to propose a new objective for training over behavior preferences, makes the learning process easier for a robot and achieves state-of-the-art results in imitation learning. It also enabled the training of a robot that can solve the tasks of opening a door and moving a pen between its fingers in simulation, a first in imitation learning with expert observations alone. The incorporation of preferences has also seen success in language modeling, where chatbots such as ChatGPT (opens in new tab)<\/span><\/a> are improving themselves by learning a reward function inferred via preferences over several samples of model responses in addition to learning from desired human conversational data.<\/p>\n\n\n\n

\n
\n
\n\t
\n\t\t
\n\t\t\t\t\t\tPublication<\/span>\n\t\t\tA Ranking Game for Imitation Learning<\/span> <\/span><\/a>\t\t\t\t\t<\/div>\n\t<\/article>\n<\/div>\n<\/div>\n\n\n\n
\n
\n\t
\n\t\t
\n\t\t\t\t\t\tDOWNLOAD<\/span>\n\t\t\tSOTA algorithms for imitation learning<\/span> <\/span><\/a>\t\t\t\t\t<\/div>\n\t<\/article>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n

Robotics has found a place in controlled environments where the tasks at hand are well-defined and repeatable, such as on a factory floor. Our framework has the potential to help enable robot learning of tasks in more dynamic environments, such as helping people with daily chores around the home.<\/p>\n\n\n\n

\n
\n
\n
With \\(\\texttt{rank-game}\\), which combines learning from preferences and demonstrations via a two-player ranking-based game, robots in simulation were trained to manipulate a pen with a dexterous hand (left) and open a door with a parallel jaw gripper (right). The successful completion of these tasks marked a first in imitation learning with expert observations alone.<\/figcaption><\/figure><\/p>\n\n\n\n

A ranking game for imitation learning<\/h2>\n\n\n\n

Inverse reinforcement learning (IRL) is a popular and effective method for imitation learning. IRL learns by inferring the reward function, also referred to as the intent of the expert<\/em>, and a policy, which specifies what actions the agent\u2014or, in our case, the robot\u2014should take in a given state to successfully mimic the expert.<\/p>\n\n\n\n

Notation:<\/em> We use \\(\\pi\\) and \\(\\pi^E\\) to denote the policy of the agent and the expert, respectively, and \\(R_{gt}\\) to be the reward function of the expert, which is unknown to the agent\/robot. \\(\\rho^\\pi\\) denotes the state-action\/state visitation distribution of policy \\(\\pi\\) in the environment\u2014the probabilistic collection of states the policy visits in the environment. We use \\(J(R;\\pi)\\) to denote the \\(\\textit{cumulative reward}\\), or the performance of policy \\(\\pi\\) under a reward function \\(R\\). We assume policy \\(\\pi\\) belongs to function class \\(\\Pi\\) and reward function R belongs to function class \\(\\mathcal{R}\\). <\/p>\n\n\n\n

The goal of imitation learning is to make the agent have the same performance as the expert under the expert\u2019s unknown reward function \\(R_{gt}\\). The classical IRL formulation tackles this by minimizing the imitation gap under a reward function that makes the performance gap the largest. We denote this framework by \\(\\texttt{imit-game}\\) and write it below formally:<\/p>\n\n\n\n

\\(\\texttt{imit-game}(\\pi,\\pi^E): \\text{argmin}_{\\pi\\in\\Pi}\\text{max}_{R\\in\\mathcal{R}} [\\mathbb{E}_{\\rho^E(s,a)}[R(s,a)]-\\mathbb{E}_{\\rho^\\pi(s,a)}[R(s,a)]]\\)<\/p>\n\n\n\n

Simply stated, the \\(\\texttt{imit-game}\\) tries to find a policy that has the lowest worst-case performance difference with the expert policy. This classical IRL formulation learns from expert demonstrations but provides no mechanism to incorporate learning from preferences. In our work, we ask, does IRL really need to consider the worst-case performance difference? We find that relaxing this requirement allows us to incorporate preferences.<\/p>\n\n\n\n

Our proposed method treats imitation as a two-player ranking-based game between a policy and a reward. In this game, the reward agent learns to map more preferred behaviors to a higher total reward for each of the pairwise preferences, while the policy agent learns to maximize the performance on this reward function by interacting with the environment. Contrary to the classical IRL framework, the reward function now has to get only the rankings correct and not optimize for the worst case (see Figure 1).<\/p>\n\n\n\n

\"A
Figure 1:<\/strong> The proposed \\(\\texttt{rank-game}\\) method treats imitation learning as a two-player ranking-based game between a policy and a reward. The policy agent maximizes the reward function by interacting with the environment. The reward agent satisfies a set of behavior rankings obtained from various sources: generated by the policy agent, automatically generated via data augmentation, or expert-annotated rankings obtained from a human or offline dataset.<\/figcaption><\/figure>\n\n\n\n

To incorporate preferences, we need to quantify the behaviors in order to compare them. In this work, we choose the behaviors (\\(\\rho\\)) to be the state-action or state-only visitation distribution of the agent. A ranking between behaviors is used to specify that the expert would prefer one behavior over the other. A reward function that satisfies the behavior rankings ensures that the average return under a lower-ranked behavior is smaller than the higher-ranked behavior. More formally, the ranking game is defined as a game where the policy agent \\(\\pi\\) maximizes the expected return \\(J(R;\\pi)\\) of the policy under reward function \\(R\\) when deployed in the environment. The reward player takes the dataset of pairwise rankings \\(D^p\\) (rankings are denoted as \\(\\rho^i\\preceq\\rho^j\\)) as an input and attempts to learn a reward function that satisfies those rankings using a ranking loss (denoted by \\(L(D^p;R)\\)).<\/p>\n\n\n\n

\\(\\underbrace{\\text{argmax}_{\\pi\\in\\Pi}J(R;\\pi)}_{\\text{Policy Agent}}~~~~~~~~~~~~~~~\\underbrace{\\text{argmin}_{R\\in\\mathcal{R}}L(D^p;R)}_{\\text{Reward Agent}}\\)<\/p>\n\n\n\n

The ranking loss induces a reward function \\(R\\) that attempts to satisfy each pairwise preference in the dataset as follows:<\/p>\n\n\n\n

\\(\\mathbb{E}_{\\rho^i}[R(s,a)]\\le\\mathbb{E}_{\\rho^j}[R(s,a)]~~,~~\\forall \\rho^i\\preceq\\rho^j \\in D^p\\)<\/p>\n\n\n\n

Generalizing prior imitation learning approaches with \\(\\texttt{rank-game}\\)<\/h2>\n\n\n\n

The \\(\\texttt{rank-game}\\) framework neatly encapsulates prior work in IRL and prior work in learning from preferences, respectively. First, let\u2019s see how classical IRL is a part of this framework. Recall that the classical IRL\/\\(\\texttt{imit-game}\\) optimization can be written as:<\/p>\n\n\n\n

\\(\\text{argmin}_{\\pi\\in\\Pi}\\text{max}_{R\\in\\mathcal{R}} [\\mathbb{E}_{\\rho^E(s,a)}[R(s,a)]-\\mathbb{E}_{\\rho^\\pi(s,a)}[R(s,a)]]\\)<\/p>\n\n\n\n

The inner optimization learns a reward function that ensures that the return gap under the reward function is maximized between the current policy\u2019s behavior and the expert behavior. Thus, \\(\\texttt{imit-game}\\) can be seen to be a special case of \\(\\texttt{rank-game}\\) with: (1) a ranking dataset that prefers expert behavior more than the current agent behavior and (2) a form of ranking loss that maximizes the performance gap (termed as \\(\\textit{supremum loss}\\)). A number of prior methods in the imitation learning domain can be understood as special cases of \\(\\texttt{rank-game}\\) under various ranking losses, classes of reward functions, and abilities to incorporate preferences (see Figure 2).<\/p>\n\n\n\n

\"A
Figure 2:<\/strong> Previous methods that learn from expert demonstrations or preferences form a special case of \\(\\texttt{rank-game}\\) under a specific choice of ranking loss and a reward function class. Also noted in the table is whether a method enables learning from demonstration (LfD)\u2014that is, learning from both expert states and actions\u2014or learning from observations (LfO), where an agent learns from expert states alone.<\/figcaption><\/figure>\n\n\n\n

Setting up the ranking game<\/h2>\n\n\n\n

To develop a framework that successfully combines learning from demonstrations and learning from preferences, we addressed several questions:<\/p>\n\n\n\n

    \n
  1. What is the ranking loss function that allows for the reward to satisfy the preferences in the dataset?<\/li>\n\n\n\n
  2. Where do we get the dataset of pairwise preferences?<\/li>\n\n\n\n
  3. How can we effectively optimize this two-player game?<\/li>\n<\/ol>\n\n\n\n

    Step 1: A new ranking loss function for reward learning<\/h3>\n\n\n\n

    Our proposed framework requires learning a reward function such that the rankings in the dataset are satisfied. While several loss functions exist in prior literature to enable this, such as Luce (opens in new tab)<\/span><\/a> Shepard (opens in new tab)<\/span><\/a>, Lov\u00e1sz-Bregman (opens in new tab)<\/span><\/a> divergences (opens in new tab)<\/span><\/a>, and the earlier discussed supremum loss, we introduce a new loss function:<\/p>\n\n\n\n

    \\(L_k(\\mathcal{D}^p;R) = \\mathbb{E}_{(\\rho^{\\pi^i},\\rho^{\\pi^j})\\sim \\mathcal{D}^p} \\Big[\\mathbb{E}_{s,a\\sim\\rho^{\\pi^i}}{[(R(s,a)-0)^2]} + \\mathbb{E}_{s,a\\sim\\rho^{\\pi^j}}{[(R(s,a)-k)^2]}\\Big]\\)<\/p>\n\n\n\n

    The loss function is simple and intuitive: For all the preference pairs in the dataset, the less preferred behavior is regressed to a return of 0 and more preferred behavior is regressed to a return of user-defined parameter \\(k\\). This loss function allows us to learn a reward function with user-defined scale \\(k\\), which plays an important role in enabling better policy optimization; it\u2019s principled and facilitates near-optimal imitation learning; and by design, it allows us to incorporate preferences.<\/p>\n\n\n\n

    Step 2: Getting the ranking dataset<\/h3>\n\n\n\n

    Besides giving more information about the expert\u2019s intent and being easy to obtain, another benefit of preferences is that they can also help learn a more informative, or shaped<\/em>, reward function. This form of reward shaping can provide better guidance for policy optimization, reducing the burden of exploring the environment to find the optimal policy and increasing sample efficiency for IRL. Our initial ranking dataset is generated by the policy agent from its interactions with the environment; we always prefer expert\u2019s behavior to be better or equal to current policy\u2019s behavior in the rankings. To further leverage the benefits of preferences, we consider two methods for augmenting this ranking dataset:<\/p>\n\n\n\n