{"id":648279,"date":"2020-04-09T11:00:23","date_gmt":"2020-04-09T18:00:23","guid":{"rendered":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/?p=648279"},"modified":"2020-04-09T11:00:23","modified_gmt":"2020-04-09T18:00:23","slug":"a-deep-generative-model-trifecta-three-advances-that-work-towards-harnessing-large-scale-power","status":"publish","type":"post","link":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/blog\/a-deep-generative-model-trifecta-three-advances-that-work-towards-harnessing-large-scale-power\/","title":{"rendered":"A deep generative model trifecta: Three advances that work towards harnessing large-scale power"},"content":{"rendered":"
<\/div>\n
\"An

Figure 1: A brief evolution of deep generative models over time, measured by model size (number of parameters) and scientific impact (number of citations to date). Three popular deep generative model types are considered: Auto-regressive models (neural language models or NLMs) in blue, Variational Autoencoders (VAEs) in green, and Generative Adversarial Networks (GANs) in orange. Transformer and BERT are shown as references. The three new generative models we introduce in this post expand large-scale capabilities in each of these categories (right side of chart).<\/p><\/div>\n

One of the core aspirations in artificial intelligence is to develop algorithms and techniques that endow computers with an ability to synthesize the observed data in our world. Every time researchers build a model to imitate this ability, this model is called a generative model. If deep neural networks are involved in this model, the model is a deep generative model (DGM). As a branch of self-supervised learning techniques in deep learning, DGMs specifically focus on characterizing data generation processes.<\/p>\n

This post describes three projects that share a common theme: improving or applying DGMs in the era of large-scale datasets and training. In this post, we\u2019ll first review the evolution history of DGMs, then introduce new advances in DGMs made by researchers from Microsoft Research, in collaboration with members from the University at Buffalo (opens in new tab)<\/span><\/a> and Duke University (opens in new tab)<\/span><\/a>. The generative models presented here, and detailed in their corresponding papers, each fall into a different category of popular deep generative model. Optimus is the first large-scale Variational Autoencoder (opens in new tab)<\/span><\/a> (VAE) language model, showing the opportunity of DGMs following a trend of pre-trained language models. FQ-GAN resolves scalability issues with image generation in Generative Adversarial Networks (opens in new tab)<\/span><\/a> (GANs). Finally, we introduce Prevalent, the first pre-trained generic agent for vision-and-language navigation<\/a>. Let\u2019s look at a quick overview of DGMs before diving into our new achievements.<\/p>\n

Three types of generative models and a shared trick<\/h3>\n

Generative models (opens in new tab)<\/span><\/a> have a long history in traditional machine learning, and they are often distinguished from the other main approach, discriminative models (opens in new tab)<\/span><\/a>. One may learn how they differ from a story of two siblings (opens in new tab)<\/span><\/a>. In the story, the siblings have different special abilities: one has the ability to learn everything in great depth, while the other can only learn the differences between what he sees. These siblings represent a generative model, which characterizes actual distributions with an internal mechanism, and a discriminative model, which builds decision boundaries between classes.<\/p>\n

With the rise of deep learning, a new family of methods, called deep generative models (DGMs), is formed through the combination of generative models and deep neural networks. Because neural networks used as generative models have a number of parameters smaller than the amount of data they are trained on, there is a trick that DGMs can perform. In an excellent blog post from OpenAI (opens in new tab)<\/span><\/a>, this trick is revealed: \u201c\u2026models are forced to discover and efficiently internalize the essence of the data in order to generate it.<\/strong>\u201d<\/em> To learn more about DGMs, check out recent lectures at Stanford University (opens in new tab)<\/span><\/a>, University of California Berkeley (opens in new tab)<\/span><\/a>, Columbia University (opens in new tab)<\/span><\/a> and New York University (opens in new tab)<\/span><\/a>.<\/p>\n

Mathematically, for a dataset of examples {\\(x_{i}\\)|\\(x_{i}\\) \\( \\in \\) \\(\\mathbb{R}\\)<sup><em>D<\/em><\/sup>, \\(i\\) = 1, … , \\(N\\)} as samples from a true data distribution \\( q(x)\\), the goal of a DGM is to build deep neural networks with parameters \\(\\theta\\) \\( \\in \\) \\(\\mathbb{R}\\)<sup><em>P<\/em><\/sup>, to describe a distribution \\(p_\\theta\\left(x\\right) \\) so that the parameters \\(\\theta\\) can be trained to ensure \\(p_\\theta\\left(x\\right) \\) match \\(q(x)\\) the best. All DGMs share this same basic setup and the above DGM trick, but they differ in the ways they approach the problem. There are three popular model types according to OpenAI taxonomy (opens in new tab)<\/span><\/a>: VAEs<\/span>, GANs<\/span>, and auto-regressive models<\/span>. Each of these is detailed in the following table:<\/p>\n

\"\"<\/p>\n

Moving from small-scale to large-scale deep generative models in all three categories<\/h3>\n

Thanks to many efforts on developing their theoretical principles over the years, DGMs are now relatively well understood at a small scale. The DGM trick mentioned above promises that the models work fine under a mild condition: \\(P\\)< \\(N\\)*\\(D\\). This has been verified in many early works at a small scale. However, recent years have witnessed tremendous advances and strong empirical results through pre-training large models on massive data (in the context of the equation above,\\(N\\)is increased dramatically).<\/p>\n

Researchers from OpenAI believe that generative models are one of the most promising approaches to potentially reach the goal of endowing computers with an understanding of our world. Along these lines, they developed Generative Pre-training (opens in new tab)<\/span><\/a> (GPT) in 2018, an autoregressive neural language model (opens in new tab)<\/span><\/a> (NLM) trained on a diverse corpus of unlabeled text, followed by discriminative fine-tuning on each specific task, showing significantly improved performance on multiple language understanding tasks. In 2019, they further scaled this idea up to 1.5 billion parameters and developed GPT-2 (opens in new tab)<\/span><\/a>, which shows near-human performance in language generation. With more compute, Megatron (opens in new tab)<\/span><\/a> and Turing-NLG<\/a> inherit the same idea, and scale it up to 8.3 billion and 17 billion, respectively.<\/p>\n

The above line of research shows that NLM has gained tremendous progress (\\(P\\) is increased dramatically in the equation above). Nevertheless, as an autoregressive model, NLM is just one of three types of DGMs. There are still two other types of DGMs (VAE and GAN) that can be significantly improved for large-scale uses. In this exciting era, large models are trained on large datasets with massive computing, which has given rise to a new learning paradigm: self-supervised pre-training with task-specific fine-tuning. DGMs have been studied less in this setting, and we are not sure if the tricks of DGMs can still work well in this setting for industrial practice. It raises a series of research questions, which we\u2019ll explore in relation to each project below:<\/p>\n