{"id":615636,"date":"2019-10-23T15:00:16","date_gmt":"2019-10-23T22:00:16","guid":{"rendered":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/?p=615636"},"modified":"2019-10-24T15:24:31","modified_gmt":"2019-10-24T22:24:31","slug":"from-blank-canvas-unfolds-a-scene-gan-based-model-generates-and-modifies-images-based-on-continual-linguistic-instruction","status":"publish","type":"post","link":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/blog\/from-blank-canvas-unfolds-a-scene-gan-based-model-generates-and-modifies-images-based-on-continual-linguistic-instruction\/","title":{"rendered":"From blank canvas unfolds a scene: GAN-based model generates and modifies images based on continual linguistic instruction"},"content":{"rendered":"

\"Illustration<\/a><\/p>\n

When people create, it\u2019s not very often they achieve what they\u2019re looking for on the first try. Creating\u2014whether it be a painting, a paper, or a machine learning model\u2014is a process that has a starting point from which new elements and ideas are added and old ones are modified and\u00a0discarded, sometimes\u00a0again and again,\u00a0until the work accomplishes\u00a0its intended purpose:\u00a0to\u00a0evoke emotion,\u00a0to\u00a0convey a message,\u00a0to\u00a0complete a task. Since I began my work as a researcher, machine learning systems have gotten really good at a particular form of creation that\u00a0has\u00a0caught my attention: image generation.<\/p>\n

Looking at some of the images\u00a0generated\u00a0by systems\u00a0such\u00a0as\u00a0BigGAN (opens in new tab)<\/span><\/a>\u00a0and\u00a0ProGAN (opens in new tab)<\/span><\/a>, you wouldn\u2019t be able to tell they were produced by a computer. In these\u00a0advancements,\u00a0my\u00a0colleagues and I\u00a0see\u00a0an opportunity to help\u00a0people\u00a0create visuals and\u00a0better\u00a0express themselves through the medium\u2014from\u00a0improving the user experience when it comes to designing avatars in the gaming world to making the editing of personal photos and production of digital art in software like Photoshop, which can be challenging to those unfamiliar with such programs\u2019 capabilities, easier.\u00a0Because of our background in dialogue, we see that help happening via natural language. We envision conversational technology that allows people to create\u00a0images\u00a0just\u00a0by\u00a0talking\u00a0or typing a series of\u00a0directions\u00a0and feedback\u00a0across multiple iterations.\u00a0We\u00a0even\u00a0think it\u2019s possible for such a system to\u00a0eventually\u00a0take a proactive approach, seeking clarification when instructions are\u00a0ambiguous,\u00a0essentially participating in a two-way conversation.<\/p>\n

Our\u00a0team,\u00a0comprising\u00a0researchers\u00a0from\u00a0across\u00a0Microsoft Research Montr\u00e9al,\u00a0the\u00a0Vector Institute (opens in new tab)<\/span><\/a>, and\u00a0the\u00a0Mila – Quebec AI Institute (opens in new tab)<\/span><\/a>,\u00a0recently introduced the\u00a0Generative Neural Visual Artist (GeNeVA) task (opens in new tab)<\/span><\/a>\u00a0and a\u00a0recurrent\u00a0generative adversarial network\u00a0(GAN)\u2013based model, GeNeVA-GAN, to tackle it.\u00a0In the GeNeVA task,\u00a0a\u00a0T<\/i>eller<\/i>, or\u00a0user,\u00a0gives\u00a0an\u00a0instruction to\u00a0the\u00a0D<\/i>rawer<\/i>,\u00a0our\u00a0system,\u00a0and the Drawer generates an image corresponding to the instruction. The\u00a0Teller\u00a0then continues to provide\u00a0further instructions for modifying the\u00a0resulting\u00a0image,\u00a0and the Drawer continues to generate\u00a0a\u00a0modified image.<\/p>\n

\"Figure<\/a>

Figure 1: In the Generative Neural Visual Artist (GeNeVA) task, the Drawer\u2014a generative adversarial network-based model\u2014iteratively constructs a scene based on instructions and feedback from a Teller, or user.<\/p><\/div>\n

The task at hand<\/h3>\n

Work in the field\u00a0of text-based image generation\u00a0has mainly been dominated by one-step generation, which unfortunately doesn\u2019t easily\u00a0lend itself to more complex\u00a0images\u00a0people may\u00a0be interested in creating;\u00a0let\u2019s say,\u00a0a park scene with multiple people picnicking, tossing a football,\u00a0or\u00a0participating in other activities.\u00a0You\u2019d\u00a0need a potentially large and detailed paragraph to\u00a0elicit such an output. Plus,\u00a0it doesn\u2019t allow for the creative\u00a0process\u00a0as people\u00a0naturally experience it.\u00a0Unless you\u2019ve specified things very precisely in the provided text\u2014place the object\u00a0one\u00a0<\/b>inch from the left and\u00a0two\u00a0from the top, for example\u2014you won\u2019t get exactly what you want.<\/p>\n

The GeNeVA task\u00a0places the focus on\u00a0iteration, testing\u00a0potential\u00a0models\u00a0on a\u00a0couple\u00a0of fronts: their ability to\u00a0convert instructions\u00a0into\u00a0appropriate\u00a0image modifications and their ability to\u00a0maintain previous instructions\u00a0and image properties, such as spatial relationships,\u00a0across\u00a0versions of the image. Since real-world image data paired with instructions is not available in large quantities, we use simpler datasets for\u00a0this\u00a0task. We introduce the Iterative CLEVR\u00a0dataset\u2014i-CLEVR, for short\u2014an iterative version\u00a0of the\u00a0Compositional Language and Elementary Visual Reasoning\u00a0(CLEVR)\u00a0dataset (opens in new tab)<\/span><\/a>\u00a0in which the scenes are\u00a0created step-by-step\u00a0using\u00a0natural language\u00a0instructions. We also use the\u00a0Collaborative Drawing (CoDraw) dataset (opens in new tab)<\/span><\/a>,\u00a0which consists of clip-art scenes of children playing in a park.<\/p>\n

Recurrent GAN\u2014our approach!<\/h3>\n

With\u00a0the goal\u00a0of\u00a0ultimately\u00a0extending\u00a0the GeNeVA task to photo-realistic images, we chose to\u00a0use\u00a0a GAN\u2013based model,\u00a0as\u00a0GANs\u00a0are on the forefront of image generation in the pixel space today.<\/p>\n

While a non-pixel-based approach\u2014where the\u00a0placement of\u00a0clip art\u00a0or cutouts\u00a0of objects from real images\u00a0is predicted,\u00a0as in the task\u00a0associated with the CoDraw dataset\u2014is easier, copy and paste can lead to less\u00a0natural-looking images.\u00a0A\u00a0pixel-based approach allows for\u00a0the\u00a0expression\u00a0of\u00a0lighting differences, a variety of angles for each\u00a0object,\u00a0and other characteristics that make for\u00a0realistic\u00a0images. To\u00a0achieve\u00a0the same effect\u00a0with a\u00a0non-pixel-based approach, you\u2019d need an infinite collection of clip art\u00a0representing\u00a0these detailed distinctions.<\/p>\n

\"GeNeVA-GAN<\/a>

Figure 2: To ensure the system maintains image details across iterations and makes modifications based on the history of instructions provided, the GeNeVA-GAN architecture incorporates a gated recurrent unit (GRU)\u2013based recurrent neural network to encode the current instructions and previous instructions and a convolutional neural network encoder to create a representation of the previous image. Both representations are passed through the generator G. An auxiliary object detector is added to the discriminator D, which allows the discriminator to determine whether the instructions were followed properly.<\/p><\/div>\n

To tackle the task,\u00a0we\u00a0integrated several\u00a0other\u00a0machine learning\u00a0components\u00a0into\u00a0the\u00a0GAN\u00a0model, including\u00a0a recurrent neural network, specifically a gated recurrent unit (GRU);\u00a0a\u00a0convolutional neural network\u00a0(CNN)\u2013based image encoder;\u00a0and an auxiliary object detector.<\/p>\n

Traditional GAN models\u00a0consist of two components: a generator, which\u00a0produces\u00a0an output given some input, and a discriminator, which differentiates between the generated\u00a0data\u00a0and the ground-truth\u00a0data. To achieve the iterative approach we were seeking,\u00a0we apply the generator at each instruction, or timestep<\/em>,\u00a0and modified the\u00a0GAN\u00a0architecture to use features from the previous timestep.\u00a0Because it\u2019s\u00a0integral\u00a0for the system\u00a0to\u00a0adhere\u00a0to\u00a0the previously provided instructions, we incorporated a\u00a0hierarchical\u00a0GRU-based\u00a0recurrent neural network\u00a0to\u00a0encode not only the current instruction, but\u00a0also\u00a0the entire state of the conversation. These\u00a0representations\u00a0are then\u00a0passed on to the next step.<\/p>\n

With just the text,\u00a0though,\u00a0there is no guarantee the modified image would carry over the same properties the user just saw and responded to, the user\u2019s current instruction aside. There could be multiple\u00a0plausible\u00a0ways of generating the image\u00a0from the provided instructions. In the\u00a0CoDraw samples\u00a0of\u00a0clip-art scenes in which\u00a0two children\u00a0are\u00a0playing\u00a0in the park,\u00a0for example,\u00a0placing the girl to the left of the boy could mean many positions\u2014 right beside him, at the\u00a0very far left of him, and everywhere in between. We want the system to maintain the precise details of the previous image\u2014to remember what was already drawn and how\u2014not regenerate\u00a0the image\u00a0from scratch\u00a0every time. To ensure this consistency, we include features from the previous image\u00a0encoded\u00a0using\u00a0a CNN.<\/p>\n

To continue to preserve the\u00a0integrity of the iterative approach,\u00a0we integrate into the discriminator an auxiliary\u00a0object\u00a0detector, enabling\u00a0the discriminator\u00a0to\u00a0determine\u00a0whether the\u00a0objects in the instructions were properly\u00a0generated\u00a0in addition to determining whether the image is a quality image.<\/p>\n

\"Example<\/a>

Figure 3: Example images generated by our best GeNeVA-GAN model on the CoDraw (top row) and i-CLEVR (bottom row) datasets; shown with the provided instructions.<\/p><\/div>\n

Face-off\u2014iterative vs. non-iterative<\/h3>\n

In experiments,\u00a0our iterative approach outperforms a non-iterative GeNeVA-GAN baseline that receives all the instructions and then only generates a single final image. Metrics commonly used for evaluating GANs only consider the generated image quality and not whether the image is accurate for the provided instruction. Hence, we also propose a relationship similarity metric,\u00a0called\u00a0rsim<\/i>,\u00a0that evaluates the model\u2019s ability to place objects in a position\u00a0that aligns\u00a0with the instructions.\u00a0This new metric measures whether the left-right, front-behind relationships among objects in the ground-truth reference image are followed in the generated image. For determining objects and their locations, we train an object detector and localizer model. We also use\u00a0the\u00a0trained object detector to evaluate precision, recall, and F1\u00a0score on object detections. The performance of both the non-iterative and our best iterative model on these metrics and on\u00a0rsim<\/i>\u00a0is presented\u00a0below\u00a0in Table 1.<\/p>\n

\"Results<\/a>

Table 1: Results of the GeNeVA-GAN model on the CoDraw and i-CLEVR datasets. Precision, recall, and F1 score measure object detection performance on the generated image with regard to ground-truth labels; rsim<\/em> measures to what extent left-right, front-behind relationships between objects in the ground-truth image are followed in the generated image. The iterative GeNeVA-GAN model can build on previous context and perform better than the non-iterative baseline.<\/p><\/div>\n

For more details, please check out our paper\u00a0\u201cTell, Draw, and Repeat: Generating and Modifying Images Based on Continual Linguistic Instruction,\u201d<\/a>\u00a0which\u00a0we\u2019re presenting at the 2019\u00a0International Conference on Computer Vision\u00a0(ICCV) (opens in new tab)<\/span><\/a>. Source code to generate the CoDraw and\u00a0i-CLEVR datasets and code to train and evaluate GeNeVA-GAN models can be found on\u00a0our\u00a0project page<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"

When people create, it\u2019s not very often they achieve what they\u2019re looking for on the first try. Creating\u2014whether it be a painting, a paper, or a machine learning model\u2014is a process that has a starting point from which new elements and ideas are added and old ones are modified and\u00a0discarded, sometimes\u00a0again and again,\u00a0until the work […]<\/p>\n","protected":false},"author":39507,"featured_media":617190,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-author-ordering":[{"type":"user_nicename","value":"Shikhar Sharma","user_id":"36557"}],"msr_hide_image_in_river":0,"footnotes":""},"categories":[194471],"tags":[243801,243786,186897,243783,243810,195663,243795,201905,243792,195953,243807,243804,243798,243789],"research-area":[13562,13554,13560],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-615636","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-computer-vision","tag-biggan","tag-computer-imaging","tag-computer-vision","tag-gan-based-model","tag-general-neural-visual-artist","tag-geneva","tag-geneva-project","tag-iccv","tag-iccv-2019","tag-international-conference-on-computer-vision","tag-microsoft-research-montreal","tag-progan","tag-shikhar-sharma","tag-tell-draw-repeat","msr-research-area-computer-vision","msr-research-area-human-computer-interaction","msr-research-area-programming-languages-software-engineering","msr-locale-en_us"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[437514],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[],"related-projects":[585388],"related-events":[610425],"related-researchers":[{"type":"user_nicename","value":"Shikhar Sharma","user_id":36557,"display_name":"Shikhar Sharma","author_link":"Shikhar Sharma<\/a>","is_active":false,"last_first":"Sharma, Shikhar","people_section":0,"alias":"shsh"}],"msr_type":"Post","featured_image_thumbnail":"\"Illustration","byline":"Shikhar Sharma<\/a>","formattedDate":"October 23, 2019","formattedExcerpt":"When people create, it\u2019s not very often they achieve what they\u2019re looking for on the first try. Creating\u2014whether it be a painting, a paper, or a machine learning model\u2014is a process that has a starting point from which new elements and ideas are added and…","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/posts\/615636","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/users\/39507"}],"replies":[{"embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/comments?post=615636"}],"version-history":[{"count":15,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/posts\/615636\/revisions"}],"predecessor-version":[{"id":617211,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/posts\/615636\/revisions\/617211"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/media\/617190"}],"wp:attachment":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/media?parent=615636"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/categories?post=615636"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/tags?post=615636"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=615636"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=615636"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=615636"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=615636"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=615636"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=615636"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=615636"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=615636"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}