{"id":447303,"date":"2017-12-06T09:36:31","date_gmt":"2017-12-06T17:36:31","guid":{"rendered":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/?p=447303"},"modified":"2023-05-01T12:06:15","modified_gmt":"2023-05-01T19:06:15","slug":"hybrid-reward-architecture-fall-ms-pac-man-dr-harm-van-seijen","status":"publish","type":"post","link":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/podcast\/hybrid-reward-architecture-fall-ms-pac-man-dr-harm-van-seijen\/","title":{"rendered":"Hybrid Reward Architecture and the Fall of Ms. Pac-Man with Dr. Harm van Seijen"},"content":{"rendered":"\n<figure class=\"wp-block-image alignright size-medium\"><a href=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/people\/havansei\/\"><img loading=\"lazy\" decoding=\"async\" width=\"300\" height=\"200\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2017\/12\/BlogPost_600x400_Harm_Podcast-300x200.jpg\" alt=\"photo of Dr. Harm van Seijen\" class=\"wp-image-447321\" srcset=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2017\/12\/BlogPost_600x400_Harm_Podcast-300x200.jpg 300w, https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2017\/12\/BlogPost_600x400_Harm_Podcast.jpg 600w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><\/figure>\n\n\n\n<h2 id=\"episode-3-december-6-2017\" class=\"wp-block-heading\">Episode 3 | December 6, 2017<\/h2>\n\n\n\n<p><strong>Hybrid Reward Architecture and the Fall of Ms. Pac-Man with Dr. Harm van Seijen<\/strong><\/p>\n\n\n\n<p>If you\u2019ve ever watched King of Kong: Fistful of Quarters, you know what a big deal it is to beat a video arcade game that was designed not to lose. Most humans can\u2019t even come close. Enter Harm van Seijen, and a team of machine learning researchers from <a href=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/lab\/microsoft-research-montreal\/\">Microsoft Research Montreal<\/a>. They took on Ms. Pac-man. And won. Today we\u2019ll talk to Harm about his work in reinforcement learning, the inspiration for hybrid reward architecture, visit a few islands of tractability and get an inside look at the science behind the AI defeat of one of the most difficult video arcade games around.<\/p>\n\n\n\n<p>To find out more about Harm van Seijen and the groundbreaking work going on at <a href=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/lab\/microsoft-research-montreal\/\">Microsoft Research Montreal<\/a>, visit <a href=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/\">Microsoft.com\/research.<\/a><\/p>\n\n\n\n<section class=\"wp-block-msr-subscribe-to-podcast subscribe-to-podcast\">\n\t<div class=\"subscribe-to-podcast__inner border-top border-bottom border-width-2\">\n\t\t<h2 class=\"h5 subscribe-to-podcast__heading\">\n\t\t\tSubscribe to the <a href=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/podcast\">Microsoft Research Podcast<\/a>:\t\t<\/h2>\n\t\t<ul class=\"subscribe-to-podcast__list list-unstyled\">\n\t\t\t\t\t\t\t<li class=\"subscribe-to-podcast__list-item\">\n\t\t\t\t\t<a class=\"subscribe-to-podcast__link\" href=\"https:\/\/itunes.apple.com\/us\/podcast\/microsoft-research-a-podcast\/id1318021537?mt=2\" target=\"_blank\" rel=\"noreferrer noopener\">\n\t\t\t\t\t\t<svg class=\"subscribe-to-podcast__svg\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" fill=\"black\" viewBox=\"0 0 32 32\">  <path d=\"M7.12 0c-3.937-0.011-7.131 3.183-7.12 7.12v17.76c-0.011 3.937 3.183 7.131 7.12 7.12h17.76c3.937 0.011 7.131-3.183 7.12-7.12v-17.76c0.011-3.937-3.183-7.131-7.12-7.12zM15.817 3.421c3.115 0 5.932 1.204 8.079 3.453 1.631 1.693 2.547 3.489 3.016 5.855 0.161 0.787 0.161 2.932 0.009 3.817-0.5 2.817-2.041 5.339-4.317 7.063-0.812 0.615-2.797 1.683-3.115 1.683-0.12 0-0.129-0.12-0.077-0.615 0.099-0.792 0.192-0.953 0.64-1.141 0.713-0.296 1.932-1.167 2.677-1.911 1.301-1.303 2.229-2.932 2.677-4.719 0.281-1.1 0.244-3.543-0.063-4.672-0.969-3.595-3.907-6.385-7.5-7.136-1.041-0.213-2.943-0.213-4 0-3.636 0.751-6.647 3.683-7.563 7.371-0.245 1.004-0.245 3.448 0 4.448 0.609 2.443 2.188 4.681 4.255 6.015 0.407 0.271 0.896 0.547 1.1 0.631 0.447 0.192 0.547 0.355 0.629 1.14 0.052 0.485 0.041 0.62-0.072 0.62-0.073 0-0.62-0.235-1.199-0.511l-0.052-0.041c-3.297-1.62-5.407-4.364-6.177-8.016-0.187-0.943-0.224-3.187-0.036-4.052 0.479-2.323 1.396-4.135 2.921-5.739 2.199-2.319 5.027-3.543 8.172-3.543zM16 7.172c0.541 0.005 1.068 0.052 1.473 0.14 3.715 0.828 6.344 4.543 5.833 8.229-0.203 1.489-0.713 2.709-1.619 3.844-0.448 0.573-1.537 1.532-1.729 1.532-0.032 0-0.063-0.365-0.063-0.803v-0.808l0.552-0.661c2.093-2.505 1.943-6.005-0.339-8.296-0.885-0.896-1.912-1.423-3.235-1.661-0.853-0.161-1.031-0.161-1.927-0.011-1.364 0.219-2.417 0.744-3.355 1.672-2.291 2.271-2.443 5.791-0.348 8.296l0.552 0.661v0.813c0 0.448-0.037 0.807-0.084 0.807-0.036 0-0.349-0.213-0.683-0.479l-0.047-0.016c-1.109-0.885-2.088-2.453-2.495-3.995-0.244-0.932-0.244-2.697 0.011-3.625 0.672-2.505 2.521-4.448 5.079-5.359 0.547-0.193 1.509-0.297 2.416-0.281zM15.823 11.156c0.417 0 0.828 0.084 1.131 0.24 0.645 0.339 1.183 0.989 1.385 1.677 0.62 2.104-1.609 3.948-3.631 3.005h-0.015c-0.953-0.443-1.464-1.276-1.475-2.36 0-0.979 0.541-1.828 1.484-2.328 0.297-0.156 0.709-0.235 1.125-0.235zM15.812 17.464c1.319-0.005 2.271 0.463 2.625 1.291 0.265 0.62 0.167 2.573-0.292 5.735-0.307 2.208-0.479 2.765-0.905 3.141-0.589 0.52-1.417 0.667-2.209 0.385h-0.004c-0.953-0.344-1.157-0.808-1.553-3.527-0.452-3.161-0.552-5.115-0.285-5.735 0.348-0.823 1.296-1.285 2.624-1.291z\"\/><\/svg>\n\t\t\t\t\t\t<span class=\"subscribe-to-podcast__link-text\">Apple Podcasts<\/span>\n\t\t\t\t\t<\/a>\n\t\t\t\t<\/li>\n\t\t\t\n\t\t\t\t\t\t\t<li class=\"subscribe-to-podcast__list-item\">\n\t\t\t\t\t<a class=\"subscribe-to-podcast__link\" href=\"https:\/\/subscribebyemail.com\/www.blubrry.com\/feeds\/microsoftresearch.xml\" target=\"_blank\" rel=\"noreferrer noopener\">\n\t\t\t\t\t\t<svg class=\"subscribe-to-podcast__svg\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" fill=\"none\" viewBox=\"0 0 32 32\"><path fill=\"currentColor\" d=\"M6.4 6a2.392 2.392 0 00-2.372 2.119L16 15.6l11.972-7.481A2.392 2.392 0 0025.6 6H6.4zM4 10.502V22.8a2.4 2.4 0 002.4 2.4h19.2a2.4 2.4 0 002.4-2.4V10.502l-11.365 7.102a1.2 1.2 0 01-1.27 0L4 10.502z\"\/><\/svg>\n\t\t\t\t\t\t<span class=\"subscribe-to-podcast__link-text\">Email<\/span>\n\t\t\t\t\t<\/a>\n\t\t\t\t<\/li>\n\t\t\t\n\t\t\t\t\t\t\t<li class=\"subscribe-to-podcast__list-item\">\n\t\t\t\t\t<a class=\"subscribe-to-podcast__link\" href=\"https:\/\/subscribeonandroid.com\/www.blubrry.com\/feeds\/microsoftresearch.xml\" target=\"_blank\" rel=\"noreferrer noopener\">\n\t\t\t\t\t\t<svg class=\"subscribe-to-podcast__svg\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" fill=\"none\" viewBox=\"0 0 32 32\"><path fill=\"currentColor\" d=\"M12.414 4.02c-.062.012-.126.023-.18.06a.489.489 0 00-.12.675L13.149 6.3c-1.6.847-2.792 2.255-3.18 3.944h13.257c-.388-1.69-1.58-3.097-3.179-3.944l1.035-1.545a.489.489 0 00-.12-.675.492.492 0 00-.675.135l-1.14 1.68a7.423 7.423 0 00-2.55-.45c-.899 0-1.758.161-2.549.45l-1.14-1.68a.482.482 0 00-.494-.195zm1.545 3.824a.72.72 0 110 1.44.72.72 0 010-1.44zm5.278 0a.719.719 0 110 1.44.719.719 0 110-1.44zM8.44 11.204A1.44 1.44 0 007 12.644v6.718c0 .795.645 1.44 1.44 1.44.168 0 .33-.036.48-.09v-9.418a1.406 1.406 0 00-.48-.09zm1.44 0V21.76c0 .793.646 1.44 1.44 1.44h10.557c.793 0 1.44-.647 1.44-1.44V11.204H9.878zm14.876 0c-.169 0-.33.035-.48.09v9.418c.15.052.311.09.48.09a1.44 1.44 0 001.44-1.44v-6.719a1.44 1.44 0 00-1.44-1.44zM11.8 24.16v1.92a1.92 1.92 0 003.84 0v-1.92h-3.84zm5.759 0v1.92a1.92 1.92 0 003.84 0v-1.92h-3.84z\"\/><\/svg>\n\t\t\t\t\t\t<span class=\"subscribe-to-podcast__link-text\">Android<\/span>\n\t\t\t\t\t<\/a>\n\t\t\t\t<\/li>\n\t\t\t\n\t\t\t\t\t\t\t<li class=\"subscribe-to-podcast__list-item\">\n\t\t\t\t\t<a class=\"subscribe-to-podcast__link\" href=\"https:\/\/open.spotify.com\/show\/4ndjUXyL0hH1FXHgwIiTWU\" target=\"_blank\" rel=\"noreferrer noopener\">\n\t\t\t\t\t\t<svg class=\"subscribe-to-podcast__svg\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" fill=\"none\" viewBox=\"0 0 32 32\"><path fill=\"currentColor\" d=\"M16 4C9.383 4 4 9.383 4 16s5.383 12 12 12 12-5.383 12-12S22.617 4 16 4zm5.08 17.394a.781.781 0 01-1.086.217c-1.29-.86-3.477-1.434-5.303-1.434-1.937.002-3.389.477-3.403.482a.782.782 0 11-.494-1.484c.068-.023 1.71-.56 3.897-.562 1.826 0 4.365.492 6.171 1.696.36.24.457.725.217 1.085zm1.56-3.202a.895.895 0 01-1.234.286c-2.338-1.457-4.742-1.766-6.812-1.747-2.338.02-4.207.466-4.239.476a.895.895 0 11-.488-1.723c.145-.041 2.01-.5 4.564-.521 2.329-.02 5.23.318 7.923 1.995.419.26.547.814.286 1.234zm1.556-3.745a1.043 1.043 0 01-1.428.371c-2.725-1.6-6.039-1.94-8.339-1.942h-.033c-2.781 0-4.923.489-4.944.494a1.044 1.044 0 01-.474-2.031c.096-.023 2.385-.55 5.418-.55h.036c2.558.004 6.264.393 9.393 2.23.497.292.663.931.371 1.428z\"\/><\/svg>\n\t\t\t\t\t\t<span class=\"subscribe-to-podcast__link-text\">Spotify<\/span>\n\t\t\t\t\t<\/a>\n\t\t\t\t<\/li>\n\t\t\t\n\t\t\t\t\t\t\t<li class=\"subscribe-to-podcast__list-item\">\n\t\t\t\t\t<a class=\"subscribe-to-podcast__link\" href=\"https:\/\/www.blubrry.com\/feeds\/microsoftresearch.xml\" target=\"_blank\" rel=\"noreferrer noopener\">\n\t\t\t\t\t\t<svg class=\"subscribe-to-podcast__svg\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" fill=\"none\" viewBox=\"0 0 32 32\"><path fill=\"currentColor\" d=\"M6.667 4a2.676 2.676 0 00-2.612 2.13v.003c-.036.172-.055.35-.055.534v18.666c0 .183.019.362.055.534v.003a2.676 2.676 0 002.076 2.075h.002c.172.036.35.055.534.055h18.666A2.676 2.676 0 0028 25.333V6.667a2.676 2.676 0 00-2.13-2.612h-.003A2.623 2.623 0 0025.333 4H6.667zM8 8h1.333C17.42 8 24 14.58 24 22.667V24h-2.667v-1.333c0-6.618-5.382-12-12-12H8V8zm0 5.333h1.333c5.146 0 9.334 4.188 9.334 9.334V24H16v-1.333A6.674 6.674 0 009.333 16H8v-2.667zM10 20a2 2 0 11-.001 4.001A2 2 0 0110 20z\"\/><\/svg>\n\t\t\t\t\t\t<span class=\"subscribe-to-podcast__link-text\">RSS Feed<\/span>\n\t\t\t\t\t<\/a>\n\t\t\t\t<\/li>\n\t\t\t\t\t<\/ul>\n\t<\/div>\n<\/section>\n\n\n\n<h3 id=\"transcript\" class=\"wp-block-heading\">Transcript<\/h3>\n\n\n\n<p><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Harm van Siejen: Rather than <\/span><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">trying to mimic how the brain works, it\u2019s more being inspired by how the brain works. This is true for neural networks for example. They are also based on how our brain processeses information. It doesn\u2019t mean that it\u2019s an exact copy of how the brain works. That\u2019s not the goal. I mean, machines have different capabilities, so it\u2019s not so much about trying to mimic exactly the human brain rather than being inspired.<\/span><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host: You\u2019re listening to the Microsoft Research Podcast. A show that brings you closer to the cutting-edge of technology research and the scientists behind it. I\u2019m your host, Gretchen Huizinga.<\/span><\/b><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">If you ever watched King of Kong fiscal of quarters, you know what a big deal it is to beat a video arcade game that was designed not to lose. Enter Dr. Harm Van Seijen and a team of machine learning researchers from Microsoft Maluuba in Montreal. They took on Ms. Pac-Man. And won. <\/span><\/b><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Today we\u2019ll talk to Harm about his work in reinforcement learning, the inspiration for Hybrid Reward Architecture, visit a few islands of tractability and get an inside look at the science behind the AI defeat of one of the most difficult video arcade games around. <\/span><\/b><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">That and much more on this episode of the Microsoft Research Podcast. <\/span><\/b><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host: Harm, give our listeners a brief description of the kind of work you do. <\/span><\/b><\/p>\n\n\n\n<p><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Harm van Seijen:\u00a0I work on reinforcement learning which is a sub-class of machine learning that focuses on learning good behaviors. So, it\u2019s a very powerful method. You can use it in many different instances. But it\u2019s, there\u2019s still a lot of research to make sure that it can be applied in its broadest setting. So we are working on those challenges to remove those obstacles so it can be applied in a very broad way.<\/span><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host:\u00a0Why is machine learning and particularly reinforcement learning such an important field in artificial intelligence? <\/span><\/b><\/p>\n\n\n\n<p><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Harm van Seijen:\u00a0So, with more and more data, you can just build more and more complex systems. So when your systems become more complex, at a certain moment, you can just not code everything by hand anymore, but you want it to learn automatically. For example, classifying images, to build a classifier that can recognize certain objects, that if you would do that by hand, it would be very complex. So if you do it by machine learning you can automatically learn it. So this helps you to build a very complex classifier that you couldn\u2019t otherwise encode by hand. And so with reinforcement learning, it\u2019s a similar thing, but it\u2019s then about behaviors. In behaviors, it\u2019s about taking actions. So you can for example, it\u2019s using Alpha Go. You are able to build a Go player that is much stronger than humans. Because it learns automatically, you can build something that is much better than humans.<\/span><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host:\u00a0One of your big achievements recently is beating the game of Ms. Pac-Man.\u00a0Why are video games so suitable for exploring artificial intelligence? <\/span><\/b><\/p>\n\n\n\n<p><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Harm van Seijen:\u00a0Well, they are suitable because they give you a very controlled environment to test certain ideas. So if you are dealing with applications in the real world, you then have to tackle the full complexity of your problem all at once. Whereas in a game, you can play with how complex you make your problem, and it\u2019s a more controlled environment to test certain ideas. Also because you can run it faster than real time for example, so you can very quickly have a very quick turnaround time for building algorithms.<\/span><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host:\u00a0So you are running the game at a faster speed than the game normally goes? <\/span><\/b><\/p>\n\n\n\n<p><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Harm van Seijen:\u00a0Yeah, exactly. So for example, if we take the example of Go as an example, so if you would play Go in real time, it\u2019s a very slow game and maybe a game lasts an hour or two hours. If you play it in an artificial environment, you can run it much faster, so you can play many, many games in the same amount of time that you would play a single game in real time. So yeah, it can give you a big speed-up in that scenario.<\/span><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host:\u00a0But Ms. Pac-Man for example, which is a different kind of a game obviously, it\u2019s a video game, that moves super-fast to begin with. Is it the same sort of incremental speed increase relevant there? <\/span><\/b><\/p>\n\n\n\n<p><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Harm van Seijen:\u00a0Yeah, I mean, because you always want to go faster. So Ms. Pac-Man we can run much faster than real time. We can run it maybe 30 or 40 times as fast as real time. So it means that your total computation time, if in real time it would take a month, if it\u2019s 30 times as fast, it would take you a single day. So it makes a big difference.<\/span><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host:\u00a0This is where machines are more capable than we are currently in terms of speed of processing and calculation and things like that. <\/span><\/b><\/p>\n\n\n\n<p><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Harm van Seijen:\u00a0Yes. It\u2019s interesting because if you look at games, then, there are a couple of aspects where machines really has an advantage and there are a couple of aspects where a human has an advantage. So if you have a game that is challenging mainly because it requires a very high reaction time, then that would be a game where machines naturally have an advantage. Versus, on the other hand, you have games that really require long-term reasoning and those are games that are very suitable for humans but these are much harder for machines.<\/span><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host:\u00a0I\u2019m wondering if we can make them feel inferior, but I don\u2019t think they have those kinds of feelings at the current time. <\/span><\/b><\/p>\n\n\n\n<p><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Harm van Seijen:\u00a0Not yet, no.<\/span><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host:\u00a0That\u2019s what we\u2019re working for, right?\u00a0Listen, as a level set, kind of talk to me a little bit about the broader world of machine learning right now and differentiate some of the concepts. <\/span><\/b><\/p>\n\n\n\n<p><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Harm van Seijen:\u00a0Within machine learning you have different sub-problems basically. So the three big ones are supervised learning, unsupervised learning and reinforcement learning. So those are really problem definitions. For example, reinforcement learning tackles the problem of an agent that interacts with an environment. And if you compare that with deep learning for example, so deep learning is really \u2013 it\u2019s not so much a problem definition as it is a technique. So it\u2019s a particular technique to do function approximation; in particular having many different layers of neural networks for example. So it\u2019s a technique you can use on these different problem instances. So if you combine deep learning with reinforcement learning, you get something called deep reinforcement learning and it just means that you are tackling the problem of reinforcement learning using a function approximation that uses deep learning.<\/span><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host:\u00a0What is hot right now?\u00a0Or are they all \u2013 are there areas in machine learning that are really interesting to a lot of people? <\/span><\/b><\/p>\n\n\n\n<p><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Harm van Seijen:\u00a0Yes. So deep learning really had a big boom a couple of years ago. So that\u2019s like super hot right now. It had like a long-history like in the 80s, it was also popular, but then it kind of died down again. And so the most recent boom was a couple of years ago when they discovered how you could build much deeper networks and much more powerful networks. Deep learning has received a big boom very recently. And I think reinforcement learning is just on the brink of breaking through. The most recent two years, a lot of companies have become very interested in reinforcement learning as well. So I think that\u2019s the next big thing.<\/span><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host:\u00a0It seems like as you say it, it makes me think that the booms come, when a particular researcher or a group of researchers or even inventors if you will, make a breakthrough in it and then everyone pays attention. It\u2019s like hey that\u2019s new, that\u2019s interesting, let\u2019s go on that thread. Where are you with reinforcement learning in that process? Are you still on kind of the research breakthrough phase? <\/span><\/b><\/p>\n\n\n\n<p><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Harm van Seijen:\u00a0So, in terms of maturity I think it\u2019s much less mature. It\u2019s much more still in the research phase than something like deep learning. So there\u2019s still a lot of problem instances that we cannot solve yet or not solve well yet. So there are a couple of islands of tractability that within certain problem instances in reinforcement learning that we can solve. So in particular, if you consider the bandit problem which is a special case of a reinforcement learning problem. That is one that we can do very well and it applies for example in ad placement. So placing ads, showing ads on a website that can be modeled as a bandit problem. So it\u2019s being used there in real products. So there are some subset of reinforcement learning we can already use and we can do well and use in real products. But for the most part, it\u2019s still a research effort.<\/span><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host:\u00a0Right. Not infancy necessarily, but certainly not mature. <\/span><\/b><\/p>\n\n\n\n<p><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Harm van Seijen:\u00a0Yes, absolutely.<\/span><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host:\u00a0Did you say islands of tractability? <\/span><\/b><\/p>\n\n\n\n<p><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Harm van Seijen:\u00a0Yes. So just certain problem instances that we have a good handle on.<\/span><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host:\u00a0Okay, good. I think I\u2019ve been stuck on an island of intractability before, but\u2026<\/span><\/b><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host:\u00a0When we talked before, you said your work encompasses computer science and neuro science. In essence you are drawing on theories of how the human brain works.\u00a0How are you applying these theories to your work with machines? <\/span><\/b><\/p>\n\n\n\n<p><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Harm van Seijen:\u00a0I would more see it as rather than trying to mimic how the brain works, it\u2019s more being inspired by how the brain works. This is true for neural networks for example. They are also based on how our brain processes information. It doesn\u2019t mean that it\u2019s an exact copy of how the brain works. That\u2019s not the goal. I mean, machines have different capabilities so it\u2019s not so much about trying to mimic exactly the human brain rather than being inspired. That also holds for certain algorithms in reinforcement learning. It\u2019s more being inspired by how we think, decision making in human\u2019s work than trying to make an exact copy of that.<\/span><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host:\u00a0That\u2019s interesting.\u00a0Speaking of inspiration, how did you come up with the idea of Hybrid Reward Architecture? What was the inspiration behind that? <\/span><\/b><\/p>\n\n\n\n<p><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Harm van Seijen:\u00a0So the inspiration really came from how humans cooperate to build great products. For example, if you have a smart phone, it\u2019s a great piece of technology. Many, many people were involved in building it. And there is not really a single person that knows how to make a smart phone. But it\u2019s really the group of persons, all of them that each have their own expertise that know how to make a smart phone. So we wanted to build something similar where if you are trying to solve a very complex task, to have a bunch of different artificial agents that work together and each agent is focused on a different aspect of the task. So each agent has a different expertise. And then by combining those agents in a particular way, they show an overall behavior that is very intelligent.<\/span><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host:\u00a0So it\u2019s kind of the distributed expertise model of business as it were. <\/span><\/b><\/p>\n\n\n\n<p><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Harm van Seijen:\u00a0Right.<\/span><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host:\u00a0But only with artificial intelligence agents within a program? <\/span><\/b><\/p>\n\n\n\n<p><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Harm van Seijen:\u00a0Yes, that\u2019s right.<\/span><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host:\u00a0So let\u2019s talk about that for a second. Because when we were talking about HRA before, you mentioned the importance. Because you\u2019ve got these agents acting individually. You program them to do specific tasks. But then there\u2019s a necessity like there is in a business for a boss to make a decision. So explain how that works with inside this Hybrid Reward Architecture (HRA) particularly with the Ms. Pac-Man task. <\/span><\/b><\/p>\n\n\n\n<p><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Harm van Seijen:\u00a0So we want all those agents to only care about their specific problem and not worry about how to collaborate with other agents. So we have a hierarchical structure where at the bottom you have all those little agents that take care of their little problem that they need to solve. So in the case of reinforcement learning, in the case of Ms. Pac-Man, the problem that you ultimately want to solve is you want to find a behavioral policy. So you want to learn what action to take given a certain screen image. So each of those little agents computes a certain preference, given the current screen image, creates a preference over the different available actions given its specific goal. So each agent has a different goal. And a goal can be something like so in the case of Ms. Pac-Man you have all these pellets on the screen. So a specific goal would be to go to one particular pellet. So the agent that is responsible for that, whose expertise is going to that pellet, tries to create a preference over the current actions and so the action that brings it as quickly as possible to this pellet, will have the highest preference. And so all of these agents, and you have more than 150 of those, they all communicate their preferences through Q values to a top agent and then the top agent kind of combines all these preferences into a single action. In the combination, it looks not just at how many agents want to go in a certain direction, but also how important a particular agent is. To put it differently, how badly a particular agent wants to take that action.\u00a0And so certain actions as going to a pellet are less important than trying to avoid a ghost. Because if you run into a ghost, you die which is very bad. So the agent that doesn\u2019t want to run into a ghost, his preference is much, much stronger for trying to avoid that ghost than an agent responsible for going to a pellet. So the top agent looks as the number of agents that want to go in a certain direction and also how important each agent is.<\/span><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host:\u00a0So number and intensity of the recommendations. <\/span><\/b><\/p>\n\n\n\n<p><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Harm van Seijen:\u00a0Yeah. That\u2019s right.<\/span><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host:\u00a0So the \u201cI see dead people\u201d agent is going to get more attention than the \u201cI see food pellet\u201d agent. <\/span><\/b><\/p>\n\n\n\n<p><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Harm van Seijen:\u00a0It\u2019s going to get more attention if the ghost is very close. If it\u2019s far away, then it doesn\u2019t matter that much. So then its preference is not that strong.<\/span><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host:\u00a0Oh, okay. So that makes a difference. The ghost could be right around the corner\u2026 or the ghost could be\u2026<\/span><\/b><\/p>\n\n\n\n<p><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Harm van Seijen:\u00a0That is very important to take the right action. And then it has a very strong intensity. Whereas if the ghost is far away, then its intensity is much lower then, maybe, fruit as close has a higher intensity.<\/span><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host:\u00a0So on that same topic, in the paper on HRA and reinforcement learning, you said the best results were achieved when the AI agent acted egotistically or selfishly and left it to the top aggregating agent to make the best move. I mean, seriously, that sounds like my family when we were growing up. It\u2019s the kids \u2013 each arguing for their own case and then Dad finally saying okay, Gretchen you get it\u2026<\/span><\/b><\/p>\n\n\n\n<p><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Harm van Seijen:\u00a0Well yes. So, you can imagine if you have different experts that you want each expert to be really good at its particular job. So it\u2019s really only going to care about its particular job for example. And then it\u2019s the top agent that listens to all of those things and makes the final decision.<\/span><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host:\u00a0Let\u2019s talk about that for a second because that, to me&#8230; We alluded to it earlier kind of this divide and conquer, and this is kind of a new thing in terms of breaking big problems down into smaller sections, assigning them and then sort of tackling the problems that way. How is that faring in research that you are doing and in any others that are kind of moving in that direction as well? <\/span><\/b><\/p>\n\n\n\n<p><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Harm van Seijen:\u00a0So there is quite some research in trying to solve a problem using multiple agents. It\u2019s always about breaking down a problem. But it is very challenging to find an architecture that can do that in a good way. So to find a good policy, you can learn it in a stable way and efficiently. So you can build many different architectures that somehow break up a problem. But to find like a good one is actually very challenging. So that\u2019s also what we spend most of our time with on this research was finding the right way to have these agents work together.<\/span><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host:\u00a0So, if Ms. Pac-Man was kind of a big achievement, and it really was, I mean it\u2019s hard, hard, hard game. With what you did to conquer that game, where could you see it being used in the real world? Where might you try it in a more, sort of, practical application?<\/span><\/b><\/p>\n\n\n\n<p><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Harm van Seijen:\u00a0Yeah, I mean, obviously the real world is extremely complex. So anything in the real world, you want to kind of break it down. So this particular technique learning very intelligent behaviors, I mean, you can think of for example a really smart Cortana for example, a really proactive personal agent.\u00a0Because if you interact with that agent, you have to take into account actions. You have to trade off immediate versus the future, things that happened in the future, things that happened immediately. I would say the real world is really complex and you want to break it down.<\/span><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host:\u00a0So, it has transferability\u2026 just how you bring that into these other scenarios is maybe one of the next steps in the research? <\/span><\/b><\/p>\n\n\n\n<p><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Harm van Seijen:\u00a0That\u2019s right.<\/span><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host:\u00a0The other question \u2013 we talked about this before and I talked about it with quite a few people in your field. This Hybrid Reward Architecture and reinforcement learning is considered an important step towards, the path towards artificial general intelligence\u2026 as opposed artificial intelligence, which is present in a lot of things as we know it now. Talk a little bit about the difference between those two concepts. <\/span><\/b><\/p>\n\n\n\n<p><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Harm van Seijen:\u00a0Right. So the difference is really about being good in one particular problem, being really good in one particular thing versus being good in many different things at the same time and kind of combining things with each other. So humans have very good general intelligence, so they can do many different tasks. Versus right now, like a lot of AI is very specialized. So they are good in one particular thing, but nothing else. So the goal towards going to general AI is trying to create a system that just like humans can do many, many different things. It can easily switch between tasks.<\/span><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host:\u00a0Yeah. And one of the things I\u2019ve read about the goal is machines that can think and reason and communicate with humans, like humans. <\/span><\/b><\/p>\n\n\n\n<p><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Harm van Seijen:\u00a0Yeah. That\u2019s the ultimate dream, right? So if you can communicate with your computer for example, just in the same way that we are communicating now, it could make things so much easier because like the world becomes more and more complex and if you have like a device that can deal with the complexity, but at the same time you can interact with in a very easy way, that can be really powerful.<\/span><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host:\u00a0What are you working on right now? <\/span><\/b><\/p>\n\n\n\n<p><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Harm van Seijen:\u00a0So yeah, like I said, it\u2019s trying to remove obstacles, trying to build better RL that, that can be applied to bigger problems. A lot has to do with scalability, RL works really great on some restricted instances. So those islands of tractability I talked about earlier, trying to increase those islands.<\/span><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host:\u00a0You\u2019re working with a group of people.\u00a0Are there separate threads or I should say lines of inquiry that you guys are dividing and conquering on? Do you work with teams that try to work on a particular problem? <\/span><\/b><\/p>\n\n\n\n<p><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Harm van Seijen:\u00a0So if you talk about Microsoft Research Maluuba, then yes, we have different teams there working on different problems. So one team works on machine reading comprehension for example. Another team works on dialogue and then we have my team that works on reinforcement learning. So within like Microsoft Research Maluuba, we have different teams working on different things. Then within the reinforcement learning group, we also have a couple of projects that we\u2019re focusing on. We try to set high goals and those have to be tackled by groups of people. You can\u2019t solve them on your own. So we really try to think well about what are the important problems, what do we want to solve and then try to create interest among multiple people so we can actually make some progress there.<\/span><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host:\u00a0So outside of Microsoft Maluuba, this sort of broader research horizon, are there other interesting things going on with reinforcement learning? <\/span><\/b><\/p>\n\n\n\n<p><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Harm van Seijen:\u00a0So there are many active areas of research within reinforcement learning. There\u2019s only a few islands of tractability that we can tackle right now. And those are things like exploration, efficient exploration, option discovery, representation learning, generalization. So there\u2019s a whole range of different active areas of research. We are working on some of them. But not all.<\/span><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host: Harm, what inspired you to get into this? <\/span><\/b><\/p>\n\n\n\n<p><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Harm van Seijen:\u00a0I have a background in applied physics. And if you look at physics as a field, then it\u2019s a couple of hundred years old. If you compare artificial intelligence with physics, it\u2019s a really new field. It\u2019s maybe 50 years old or something. So it\u2019s a really exciting area to do research in. It could have such a big impact on our society if you can actually well solve general AI for example. If it would solve it, it would completely change our society. But even if you make steps towards that, it can already have a really big impact. So it\u2019s exciting in two ways: from a research perspective it\u2019s exciting because it\u2019s a new field compared to the different sciences and it can have a massive impact in the world. So those two things is what makes it really exciting for me.<\/span><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host:\u00a0And you were that kind of kid growing up, like what can I discover, how can I change the world? <\/span><\/b><\/p>\n\n\n\n<p><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Harm van Seijen:\u00a0I was always interested in research. So it took me a while before finding the right type of research I guess. But from the start, I\u2019ve always been very \u201cresearch\u201d I guess.<\/span><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host:\u00a0Yeah? Even as a child? <\/span><\/b><\/p>\n\n\n\n<p><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Harm van Seijen:\u00a0Yes, absolutely.<\/span><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host:\u00a0What kind of research did you do when you were young? <\/span><\/b><\/p>\n\n\n\n<p><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Harm van Seijen:\u00a0For example, different kinds of puzzles. I was always interested in different kinds of puzzles. What you do now is kind of similar things, solving puzzles, but then much harder puzzles. So really I see what I do right now is kind of similar of what I did when I was 10 years old, just at a different level.<\/span><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host:\u00a0Exponentially different\u2026 Do you see a lot of talent coming up in universities, whether it\u2019s in Europe or here or Canada that are ready to take the baton and run in this field? <\/span><\/b><\/p>\n\n\n\n<p><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Harm van Seijen:\u00a0Yeah, I mean, all across the world, I think AI is getting more and more popular at universities as well. I think here in Canada we\u2019re really at the forefront. So we have some great universities here where some of these techniques, deep learning, but also reinforcement learning that came from the universities here in Canada. It feels like the right place to be.<\/span><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host:\u00a0It sounds like you are in the right place and we\u2019re excited to watch as this field continues to grow and change the world. Harm, thanks for joining us today. <\/span><\/b><\/p>\n\n\n\n<p><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Harm van Seijen:\u00a0Yeah, thank you very much.<\/span><\/p>\n\n\n\n<p><b><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">Host:\u00a0To find out more about Dr. Harm van Seijen, and the groundbreaking work going on at Microsoft Maluuba, visit Microsoft.com\/research. <\/span><\/b><\/p>\n\n\n\n<p><b><i><span style=\"margin: 0px; color: #333333; font-family: 'Georgia',serif; font-size: 12pt;\">[End of recording]<\/span><\/i><\/b><\/p>\n","protected":false},"excerpt":{"rendered":"<p>If you\u2019ve ever watched King of Kong: Fistful of Quarters, you know what a big deal it is to beat a video arcade game that was designed not to lose. Most humans can\u2019t even come close. Enter Harm van Seijen, and a team of machine learning researchers from Microsoft Research Montreal. They took on Ms. Pac-man. And won. Today we\u2019ll talk to Harm about his work in reinforcement learning, the inspiration for hybrid reward architecture, visit a few islands of tractability and get an inside look at the science behind the AI defeat of one of the most difficult video arcade games around.<\/p>\n","protected":false},"author":37074,"featured_media":447330,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"https:\/\/player.blubrry.com\/id\/29496889\/","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-author-ordering":[],"msr_hide_image_in_river":0,"footnotes":""},"categories":[240054],"tags":[187359,186418,241068,240852,186547],"research-area":[13556],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-447303","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-msr-podcast","tag-artificial-intelligence","tag-machine-learning","tag-maluuba","tag-podcast","tag-reinforcement-learning","msr-research-area-artificial-intelligence","msr-locale-en_us"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"https:\/\/player.blubrry.com\/id\/29496889\/","podcast_episode":"","msr_research_lab":[437514],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[395930],"related-projects":[577638],"related-events":[],"related-researchers":[],"msr_type":"Post","featured_image_thumbnail":"<img width=\"480\" height=\"280\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2017\/12\/ResearchAreaCarousel_480x280_Harm_Podcast.jpg\" class=\"img-object-cover\" alt=\"photo of Dr. Harm van Seijen smiling for the camera\" decoding=\"async\" loading=\"lazy\" srcset=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2017\/12\/ResearchAreaCarousel_480x280_Harm_Podcast.jpg 480w, https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2017\/12\/ResearchAreaCarousel_480x280_Harm_Podcast-300x175.jpg 300w\" sizes=\"auto, (max-width: 480px) 100vw, 480px\" \/>","byline":"","formattedDate":"December 6, 2017","formattedExcerpt":"If you\u2019ve ever watched King of Kong: Fistful of Quarters, you know what a big deal it is to beat a video arcade game that was designed not to lose. Most humans can\u2019t even come close. Enter Harm van Seijen, and a team of machine&hellip;","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/posts\/447303","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/users\/37074"}],"replies":[{"embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/comments?post=447303"}],"version-history":[{"count":12,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/posts\/447303\/revisions"}],"predecessor-version":[{"id":937878,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/posts\/447303\/revisions\/937878"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/media\/447330"}],"wp:attachment":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/media?parent=447303"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/categories?post=447303"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/tags?post=447303"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=447303"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=447303"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=447303"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=447303"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=447303"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=447303"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=447303"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=447303"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}