{"id":591421,"date":"2019-06-12T10:05:04","date_gmt":"2019-06-12T17:05:04","guid":{"rendered":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/?p=591421"},"modified":"2019-06-12T10:11:20","modified_gmt":"2019-06-12T17:11:20","slug":"help-training-assistive-indoor-agents-to-ask-for-assistance-via-imitation-learning","status":"publish","type":"post","link":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/blog\/help-training-assistive-indoor-agents-to-ask-for-assistance-via-imitation-learning\/","title":{"rendered":"HELP! Training assistive indoor agents to ask for assistance via imitation learning"},"content":{"rendered":"

\"Training<\/a><\/p>\n

Today people use personal digital assistants for help with scheduling, playing music, turning on or adjusting other devices, and answering basic questions such as \u201cWhat time\u2019s the game on?\u201d or \u201cWhere\u2019s the nearest hardware store?\u201d But what if these assistants could do more to help us in our daily lives?<\/p>\n

Imagine it\u2019s 10 p.m., and you\u2019ve just settled in for some much-needed sleep when you\u2019re jolted awake by a single thought: Did I leave the back door open? No worries. That personal digital assistant of yours is on wheels and able to provide the answer to your question without you having to further disrupt your nighttime routine by getting out of bed. You ask the digital assistant to check the door for you, but before doing so, it asks for clarification. Which door<\/em>? You respond, and your digital assistant is off. It winds its way to the back of the house, identifies the right door, determines whether it\u2019s open, closes it if need be, and returns with an update. The open door has been closed<\/em>. The report is reassuring, and you fall asleep with peace of mind.<\/p>\n

That\u2019s one of the dream scenarios for those working in artificial intelligence. Our goal is to have robots in the physical world and agents in the digital, virtual, and mixed worlds naturally interact with people via language to assist them in a variety of tasks. We have some way to go toward that goal, since even relatively simple scenarios such as a home assistive robot physically helping us locate a misplaced cellphone\u2014oh, how useful that would be!\u2014aren\u2019t as easy as they seem.<\/p>\n

To help bring the dream within reach, we\u2019ve created Vision-based Navigation with Language-based Assistance (VNLA). VNLA is a new grounded vision-language task for training agents not only to respond to open-ended requests\u2014that is, those without turn-by-turn instructions\u2014but to also strategically ask for help via language when it is needed. This capability relies on a novel framework we\u2019ve termed \u201cImitation Learning with Indirect Intervention\u201d (I3L). We\u2019re presenting the paper on this work<\/a> at the annual computer vision conference CVPR (opens in new tab)<\/span><\/a>. A video demonstrating the approach is available (opens in new tab)<\/span><\/a>, and code and data-downloading scripts for the work can be accessed via GitHub (opens in new tab)<\/span><\/a>.<\/p>\n

What\u2019s holding AI back?<\/h3>\n

Requests such as \u201ccheck the back door\u201d and \u201chelp me find my phone\u201d pose significant challenges to AI systems today. Among them:<\/p>\n

    \n
  1. Grounding natural language to vision:<\/strong> The agent has to understand what is being asked of it. In the example of the missing cellphone, that means knowing which cellphone the person is referring to when he or she says \u201cmy,\u201d what the word \u201cphone\u201d means visually, and when the phone of interest is in its field of view.<\/li>\n
  2. Navigating and avoiding collisions in GPS-denied environments:<\/strong> The agent must also understand common locations within the house where phones are likely to be left and how to navigate efficiently to these locations without explicit localization information via simultaneous localization and mapping (SLAM) or GPS. Further, the agent\u2014just like humans\u2014must be able to do this without access to an explicit metric map of the house. It must also navigate to these locations without bumping into or colliding with people and household objects sharing the space.<\/li>\n
  3. Interacting with people more naturally:<\/strong> When people ask one another for help, communication is not limited to a single command; there is a back-and-forth, a giving and taking of information. Often when lending a hand, people will ask specific follow-up questions that can yield valuable feedback. In the case of the cellphone, perhaps they\u2019ll ask for a reminder of what color the cellphone case is or where the person last remembers using it, and they can pose these types of questions at any point in the process. We believe robots and agents must be endowed with similar abilities to collaboratively accomplish tasks. A first step would be for robots and agents to understand how to deal appropriately with cases of known unknowns. In other words, they need to understand when they’re uncertain and should ask for help.<\/li>\n<\/ol>\n

    Overcoming the hurdles<\/h3>\n

    Each of these challenges represents an active research area crucial to the advancement of AI. They\u2019re particularly important in the case of interactive robots, where the challenges are occurring simultaneously, which also exacerbates the issue of collecting sequential interaction data to train such agents. These settings are inherently non-i.i.d\u2014independent and identically distributed\u2014ensuring that na\u00efve supervised learning will fail (opens in new tab)<\/span><\/a> when fielded.<\/p>\n

    \"Figure<\/a>

    Figure 1: An example run of the Vision-based Navigation with Language-based Assistance task in an unseen environment. (a) A bird\u2019s-eye view of the environment annotated with the agent\u2019s path. The agent observes the environment only through a first-person view. (b) A requester (wearing a hat) asks the agent to find a towel in the kitchen. Two towels are in front of the agent, but the room is labeled as a \u201cbathroom.\u201d The agent ignores them without being given the room label. (c) The agent leaves the bathroom. Sensing that it is lost, the agent signals the advisor (with mustache) for help. The advisor responds with an \u201ceasier\u201d low-level subgoal: \u201cTurn 60 degrees right, go forward, turn left.\u201d (d) After executing the subgoal, the agent is closer to the kitchen, but still confused. It requests help again. (e) Executing the second subgoal helps the agent see the target towel.<\/p><\/div>\n

    In our task, we ask the agent\u2014which \u201csees\u201d its environment via a monocular camera that captures its view as an image\u2014to find objects in specific locations. For example, we may request that our agent find a towel in the kitchen as demonstrated by Figure 1. Through the solution pathway of our task, we address the challenges in several ways.<\/p>\n

    For one, we leverage rich simulation environments in helping ground language to vision. Robotics and vision communities have increasingly come to rely on rich high-fidelity simulation environments such as AirSim (opens in new tab)<\/span><\/a> for training agents end-to-end. We use the Matterport3D dataset (opens in new tab)<\/span><\/a>, which has high-fidelity 3D reconstructions of real homes via the Room-to-Room (opens in new tab)<\/span><\/a> simulator, to train our agent in a photorealistic environment.<\/p>\n

    Secondly, we chose to use imitation learning over reinforcement learning for the training paradigm. Imitation learning, in which the agent learns directly from expert demonstrations rather than a reward function used in traditional reinforcement learning, can be exponentially faster in terms of trials with the environment. One drawback, though, is that imitation learning often relies on human<\/em> experts to demonstrate the optimal sequence of actions necessary to complete the task, which can be costly. Not so here, another reason we chose to use IL. Simulation training provides a natural programmatic expert at training time at no extra cost: a planning algorithm with access to the full environment state. Specifically, we have a shortest path algorithm (A*), which has knowledge of the full map and location of all objects in the scene. This paradigm of imitating an expert that has much more information than the agent has been effectively used in complex planning problems in robotics (opens in new tab)<\/span><\/a>.<\/p>\n

    I am lost\u2014help!<\/h3>\n

    Lastly\u2014and most importantly\u2014we train the agent to ask for help. In our task, there are an additional two key players: the requester\u2014in real-world scenarios, the person giving the task\u2014and the advisor. In real-world scenarios, the person giving the task would generally also be the advisor, but in theory, the advisor can be any entity with the ability to guide the agent. During execution of the task, the agent can ask for assistance from the advisor when it thinks it\u2019s lost and can do so a budgeted number of times. A budget is important; otherwise, the agent\u2014seeking to increase its success and noting that asking for help does so\u2014will learn to ask for help at every step, which largely defeats the purpose. After all, who really wants an assistant that can\u2019t get through a task without asking a million questions? The agent receives help via language aimed at putting it back on track for completing the task. For example, the advisor may say, \u201cFrom where you are, turn right and take three steps.\u201d<\/p>\n

    This way of providing assistance, via language, seeks to mimic the natural form of communication people use to help each other. With the ability to ask for help at critical points, the agent is able to successfully complete the tasks more often. In unseen environments, it performs more than five times better than baseline agents that don\u2019t allow for such intervention. During training time, we also teach the agent when<\/em> it should ask for help. This form of learning how to strategically<\/em> ask for help improves agent performance in unseen test scenarios by about 38 percent over asking for help randomly and by about 72 percent over asking for help in the beginning. Learning when it isn\u2019t confident and will benefit from asking for help is especially important for AI agents not only because doing so would help create more natural interactions, but also because AI agents are imperfect and intervention would be useful in helping the agent safely carry out complex requests.<\/p>\n

    \"Figure<\/a>

    Figure 2: Two decoding passes of the navigation module. (a) The first decoding pass computes the tentative navigation distribution, which is used as a feature for computing the help-requesting distribution. (b) The second pass computes the final navigation distribution.<\/p><\/div>\n

    In the overall agent policy architecture, the agent runs two forward passes as shown in Figure 2. In the first pass, the tentative navigation distribution is computed and is used as a feature for the help-requesting decision. If there is a lot of uncertainty in the navigation distribution, then the agent can decide if it should stop and ask for help, so long as its budget allows. In the second pass, it computes the final navigation distribution, taking into account the extra help provided if help was requested.<\/p>\n

    Our framework seeks to help facilitate in our personal digital assistants the kinds of back-and-forth common to people helping one another. We see VNLA as a foundation on which to realize richer human-AI collaboration that incorporates more natural language and in which robots and agents can participate in perspective taking.<\/p>\n

    This work was spearheaded by University of Maryland, College Park PhD student Khanh Nguyen during a Microsoft Research summer internship. Team members Debadeepta Dey, Chris Brockett, and Bill Dolan served as advisors on the work.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

    Today people use personal digital assistants for help with scheduling, playing music, turning on or adjusting other devices, and answering basic questions such as \u201cWhat time\u2019s the game on?\u201d or \u201cWhere\u2019s the nearest hardware store?\u201d But what if these assistants could do more to help us in our daily lives? Imagine it\u2019s 10 p.m., and […]<\/p>\n","protected":false},"author":38022,"featured_media":592069,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-author-ordering":null,"msr_hide_image_in_river":0,"footnotes":""},"categories":[194466,194467],"tags":[],"research-area":[13556,13562],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-591421","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-algorithms","category-artifical-intelligence","msr-research-area-artificial-intelligence","msr-research-area-computer-vision","msr-locale-en_us"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[144633],"related-projects":[],"related-events":[591436],"related-researchers":[{"type":"guest","value":"khanh-nguyen","user_id":"591451","display_name":"Khanh Nguyen","author_link":"Khanh Nguyen<\/a>","is_active":true,"last_first":"Nguyen, Khanh","people_section":0,"alias":"khanh-nguyen"}],"msr_type":"Post","featured_image_thumbnail":"\"\"","byline":"Debadeepta Dey, Khanh Nguyen<\/a>, Chris Brockett, and Bill Dolan","formattedDate":"June 12, 2019","formattedExcerpt":"Today people use personal digital assistants for help with scheduling, playing music, turning on or adjusting other devices, and answering basic questions such as \u201cWhat time\u2019s the game on?\u201d or \u201cWhere\u2019s the nearest hardware store?\u201d But what if these assistants could do more to help…","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/posts\/591421","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/users\/38022"}],"replies":[{"embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/comments?post=591421"}],"version-history":[{"count":14,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/posts\/591421\/revisions"}],"predecessor-version":[{"id":592516,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/posts\/591421\/revisions\/592516"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/media\/592069"}],"wp:attachment":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/media?parent=591421"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/categories?post=591421"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/tags?post=591421"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=591421"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=591421"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=591421"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=591421"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=591421"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=591421"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=591421"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=591421"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}