{"id":823648,"date":"2022-03-08T09:00:26","date_gmt":"2022-03-08T17:00:26","guid":{"rendered":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/?p=823648"},"modified":"2022-08-17T09:58:51","modified_gmt":"2022-08-17T16:58:51","slug":"%c2%b5transfer-a-technique-for-hyperparameter-tuning-of-enormous-neural-networks","status":"publish","type":"post","link":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/blog\/%c2%b5transfer-a-technique-for-hyperparameter-tuning-of-enormous-neural-networks\/","title":{"rendered":"\u00b5Transfer: A technique for hyperparameter tuning of enormous neural networks"},"content":{"rendered":"\n
\"An<\/figure>\n\n\n\n

Great scientific achievements cannot be made by trial and error alone. Every launch in the space program is underpinned by centuries of fundamental research in aerodynamics, propulsion, and celestial bodies. In the same way, when it comes to building large-scale AI systems, fundamental research forms the theoretical insights that drastically reduce the amount of trial and error necessary and can prove very cost-effective. <\/p>\n\n\n\n

In this post, we relay how our fundamental research enabled us, for the first time, to tune enormous neural networks that are too expensive to train more than once. We achieved this by showing that a particular parameterization preserves optimal hyperparameters across different model sizes. This is the \u00b5-Parametrization (or \u00b5P<\/em>, pronounced \u201cmyu-P”) that we introduced in a previous paper (opens in new tab)<\/span><\/a>, where we showed that it uniquely enables maximal feature learning in the infinite-width limit. In collaboration with researchers at OpenAI (opens in new tab)<\/span><\/a>, we verified its practical advantage on a range of realistic scenarios, which we describe in our new paper, “Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer<\/a>.”<\/p>\n\n\n\n

By greatly reducing the need to guess which training hyperparameters to use, this technique can accelerate research on enormous neural networks, such as GPT-3 (opens in new tab)<\/span><\/a> and potentially larger successors in the future. We also released a PyTorch package that facilitates the integration of our technique in existing models, available on the project GitHub page (opens in new tab)<\/span><\/a> or by simply running \\(\\texttt{pip install mup}\\).<\/p>\n\n\n\n

<\/p>\n\n\n\n

\n
Read the paper<\/a><\/div>\n\n\n\n
Download the code<\/a><\/div>\n<\/div>\n\n\n\n
<\/div>\n\n\n\n

“\u00b5P provides an impressive step toward removing some of the black magic from scaling up neural networks. It also provides a theoretically backed explanation of some tricks used by past work, like the T5 model (opens in new tab)<\/span><\/a>. I believe both practitioners and researchers alike will find this work valuable.”<\/p>\u2014 Colin Raffel, Assistant Professor of Computer Science, University of North Carolina at Chapel Hill and co-creator of T5<\/cite><\/blockquote><\/figure>\n\n\n\n

Scaling the initialization is easy, but scaling training is hard<\/h2>\n\n\n\n

Large neural networks are hard to train partly because we don\u2019t understand how their behavior changes as their size increases. Early work on deep learning, such as by Glorot & Bengio (opens in new tab)<\/span><\/a> and He et al.<\/a>, generated useful heuristics that deep learning practitioners widely use today. In general, these heuristics try to keep the activation scales consistent at initialization. However, as training starts, this consistency breaks at different model widths, as illustrated on the left in Figure 1.<\/p>\n\n\n\n

Unlike at random initialization, behavior during training is much harder to mathematically analyze. Our goal is to obtain a similar consistency so that as model width increases, the change in activation scales during training stay consistent and similar to initialization to avoid numerical overflow and underflow. Our solution, \u00b5P, achieves this goal, as seen on the right in Figure 1, which shows the stability of network activation scales for the first few steps of training across increasing model width. <\/p>\n\n\n\n

\"Two<\/a>
Figure 1: In the default parameterization in PyTorch, the graph on the left, the activation scales diverge in width after one step of training. But in \u00b5P, the graph on the right, the activation scales change by a consistent amount regardless of width for any training step. The y-axis shows the change of network activation scales on a fixed input after t=0, 1, 2, 3, and 4 steps of training as the width of the model varies, which is shown along the x-axis. <\/figcaption><\/figure>\n\n\n\n

Our parameterization, which maintains this consistency during training, follows two pieces of crucial insight. First, gradient updates behave differently from random weights when the width is large. This is because gradient updates are derived from data and contain correlations, whereas random initializations do not. Therefore, they need to be scaled differently. Second, parameters of different shapes also behave differently when the width is large. While we typically divide parameters into weights and biases, with the former being matrices and the latter vectors, some weights behave like vectors in the large-width setting. For example, the embedding matrix in a language model is of size vocabsize x width<\/em>. While the width tends to infinity, vocabsize<\/em> stays constant and finite. During matrix multiplication, the difference in behavior between summing along a finite dimension and an infinite one cannot be more different.<\/p>\n\n\n\n

These insights, which we discuss in detail in a previous blog post<\/a>, motivated us to develop \u00b5P. In fact, beyond just keeping the activation scale consistent throughout training, \u00b5P ensures that neural networks of different and sufficiently large widths behave similarly during training such that they converge to<\/em> a desirable limit, which we call the feature learning limit<\/em>. <\/p>\n\n\n\n\t

\n\t\t\n\n\t\t

\n\t\tSpotlight: Event Series<\/span>\n\t<\/p>\n\t\n\t

\n\t\t\t\t\t\t
\n\t\t\t\t\n\t\t\t\t\t\"Research\n\t\t\t\t<\/a>\n\t\t\t<\/div>\n\t\t\t\n\t\t\t
\n\n\t\t\t\t\t\t\t\t\t

Microsoft Research Forum<\/h2>\n\t\t\t\t\n\t\t\t\t\t\t\t\t

Join us for a continuous exchange of ideas about research in the era of general AI. Watch the latest episodes on demand.<\/p>\n\t\t\t\t\n\t\t\t\t\t\t\t\t

\n\t\t\t\t\t
\n\t\t\t\t\t\t\n\t\t\t\t\t\t\tWatch on-demand\t\t\t\t\t\t<\/a>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t<\/div>\n\t<\/div>\n\t\n\n\n

A theory-guided approach to scaling width<\/h2>\n\n\n\n

Our theory of scaling enables a procedure to transfer training hyperparameters across model sizes. If, as discussed above, \u00b5P networks of different widths share similar training dynamics, they likely also share similar optimal hyperparameters. Consequently, we can simply apply the optimal hyperparameters of a small model directly onto a scaled-up version. We call this practical procedure \u00b5Transfer<\/em>. If our hypothesis is correct, the training loss-hyperparameter curves for \u00b5P models of different widths would share a similar minimum. <\/p>\n\n\n\n

Conversely, our reasoning suggests that no scaling rule of initialization and learning rate other than \u00b5P can achieve the same result. This is supported by the animation below. Here, we vary the parameterization by interpolating the initialization scaling and the learning rate scaling between PyTorch default and \u00b5P. As shown, \u00b5P is the only parameterization that preserves the optimal learning rate across width, achieves the best performance for the model with width 213 = 8192, and where wider models always do better for a given learning rate\u2014that is, graphically, the curves don\u2019t intersect. <\/p>\n\n\n\n

\"An<\/a>
Figure 2: On the left, we train multilayer perceptrons (MLPs) of different widths (which correspond to the curves of different colors and patterns) with different learning rates (shown along the x-axis) on CIFAR10 and plot the training loss along the y-axis. On the right, the 2D plane of parameterizations is formed by interpolation of 1) the initialization scaling between PyTorch default and \u00b5P (x-axis), and 2) the learning rate scaling between PyTorch default and \u00b5P (y-axis). On this plane, PyTorch default is represented by (0, 0) and \u00b5P by (1, 1). The width-256 (log2(width) = 8) model is the same across all frames (except for random seed), but we widen models according to the parameterization represented by the dot on the right. <\/figcaption><\/figure>\n\n\n\n

Building on the theoretical foundation of Tensor Programs<\/a>, \u00b5Transfer works automatically for advanced architectures, such as Transformer (opens in new tab)<\/span><\/a> and ResNet<\/a>. It can also simultaneously transfer a wide range of hyperparameters. Using Transformer as an example, we demonstrate in Figure 3 how the optima of key hyperparameters are stable across widths. <\/p>\n\n\n\n

\"Four<\/a>
Figure 3: Transformers of different widths parameterized in \u00b5P and trained on WikiText-2<\/a>. As we increase model width, the optimal learning rate, cross-entropy temperature, initialization scale, and learning rate schedule remain stable. We can meaningfully predict the optimal hyperparameters of a wider network by looking at those of a narrow one. In plot on the lower right, we tried the following learning rate schedules: (a) linear decay, (b) StepLR @ [5k, 8k] with a decay factor of 0.1, (c) StepLR @ [4k, 7k] with a decay factor of 0.3, (d) cosine annealing,(e) constant, and (f) inverse square-root decay. <\/figcaption><\/figure>\n\n\n\n

\u201cI am excited about \u00b5P advancing our understanding of large models. \u00b5P’s principled way of parameterizing the model and selecting the learning rate make it easier for anybody to scale the training of deep neural networks. Such an elegant combination of beautiful theory and practical impact.\u201d<\/p>\u2014 Johannes Gehrke, Technical Fellow, Lab Director of Research at Redmond, and CTO and Head of Machine Learning for the Intelligent Communications and Conversations Cloud (IC3)<\/cite><\/blockquote><\/figure>\n\n\n\n

Beyond width: Empirical scaling of model depth and more<\/h2>\n\n\n\n

Modern neural network scaling involves many more dimensions than just width. In our work, we also explore how \u00b5P can be applied to realistic training scenarios by combining it with simple heuristics for nonwidth dimensions. In Figure 4, we use the same transformer setup to show how the optimal learning rate remains stable within reasonable ranges of nonwidth dimensions. For hyperparameters other than learning rate, see Figure 19 in our paper. <\/p>\n\n\n\n

\"Four<\/a>
Figure 4: Transformers of different sizes parameterized in \u00b5P and trained on Wikitext-2<\/a>. Not only does the optimal learning rate transfer across width, as shown in Figure 3, it also empirically transfers across other scale dimensions\u2014such as depth, batch size, and sequence length\u2014across the ranges we tested here. This means we can combine our theoretically motivated transfer across width with the empirically verified one across other scale dimensions to obtain the practical procedure, \u00b5Transfer, to tune hyperparameters indirectly on a small model and transfer to a large one. <\/figcaption><\/figure>\n\n\n\n

Testing \u00b5Transfer<\/h2>\n\n\n\n

Now that we have verified the transfer of individual hyperparameters, it is time to combine them in a more realistic scenario. In Figure 5, we compare \u00b5Transfer, which transfers tuned hyperparameters from a small proxy model, with directly tuning the large target model. In both cases, the tuning is done via random search. Figure 5 illustrates a Pareto frontier (opens in new tab)<\/span><\/a> of the relative tuning compute budget compared with the tuned model quality (BLEU score) on IWSLT14 De-En (opens in new tab)<\/span><\/a>, a machine translation dataset. Across all compute budget levels, \u00b5Transfer is about an order of magnitude (in base 10) more compute-efficient for tuning. We expect this efficiency gap to dramatically grow as we move to larger target model sizes. <\/p>\n\n\n\n

\"A<\/a>
Figure 5: Across different tuning budgets, \u00b5Transfer dominates the baseline method of directly tuning the target model. As we train larger target models with billions of parameters, we expect the performance gap to widen, since the proxy model can remain small while still meaningfully predicting the optimal hyperparameters, as shown in Figures 3 and 4. <\/figcaption><\/figure>\n\n\n\n

A glimpse of the future: \u00b5P + GPT-3<\/h2>\n\n\n\n

Before this work, the larger a model was, the less well-tuned we expected it to be due to the high cost of tuning. Therefore, we expected that the largest models could benefit the most from \u00b5Transfer, which is why we partnered with OpenAI to evaluate it on GPT-3. <\/p>\n\n\n\n

After parameterizing a version of GPT-3 with relative attention in \u00b5P, we tuned a small proxy model with 40 million parameters before copying the best hyperparameter combination to the 6.7-billion parameter variant of GPT-3, as prescribed by \u00b5Transfer. The total compute used during this tuning stage was only 7 percent of the compute used in the pretraining of the final 6.7-billion model. This \u00b5Transferred model outperformed the model of the same size (with absolute attention) in the original GPT-3 paper (opens in new tab)<\/span><\/a>. In fact, it performs similarly to the model (with absolute attention) with double the parameter count from the same paper, as shown in Figure 6. <\/p>\n\n\n\n

\"Two<\/a>
Figure 6: We applied \u00b5Transfer to GPT-3 6.7-billion parameter model with relative attention and obtained better results than the baseline with absolute attention used in the original GPT-3 paper<\/a>, all while only spending 7 percent of the pretraining compute budget on tuning. The performance of this \u00b5Transfer 6.7-billion model is comparable to that of the 13-billion model (with absolute attention) in the original GPT-3 paper.<\/figcaption><\/figure>\n\n\n\n

Implications for deep learning theory<\/h2>\n\n\n\n

As shown previously, \u00b5P gives a scaling rule which uniquely preserves the optimal hyperparameter combination across models of different widths in terms of training loss. Conversely, other scaling rules, like the default in PyTorch or the NTK parameterization (opens in new tab)<\/span><\/a> studied in the theoretical literature, are looking at regions in the hyperparameter space farther and farther from the optimum as the network gets wider. In that regard, we believe that the feature learning limit of \u00b5P, rather than the NTK limit (opens in new tab)<\/span><\/a>, is the most natural limit to study if our goal is to derive insights that are applicable to feature learning neural networks used in practice. As a result, more advanced theories on overparameterized neural networks should reproduce the feature learning limit of \u00b5P in the large width setting. <\/p>\n\n\n\n

Theory of Tensor Programs<\/h2>\n\n\n\n

The advances described above are made possible by the theory of Tensor Programs<\/a> (TPs) developed over the last several years. Just as autograd (opens in new tab)<\/span><\/a> helps practitioners compute the gradient of any general computation graph, TP theory enables researchers to compute the limit of any general computation graph when its matrix dimensions become large. Applied to the underlying graphs for neural network initialization, training, and inference, the TP technique yields fundamental theoretical results, such as the architectural universality of the Neural Network-Gaussian Process correspondence<\/a> and the Dynamical Dichotomy theorem<\/a>, in addition to deriving \u00b5P and the feature learning limit<\/a> that led to \u00b5Transfer. Looking ahead, we believe extensions of TP theory to depth, batch size, and other scale dimensions hold the key to the reliable scaling of large models beyond width. <\/p>\n\n\n\n

Applying \u00b5Transfer to your own models<\/h2>\n\n\n\n

Even though the math can be intuitive, we found that implementing \u00b5P (which enables \u00b5Transfer) from scratch can be error prone. This is similar to how autograd is tricky to implement from scratch even though the chain rule for taking derivatives is very straightforward. For this reason, we created the mup package (opens in new tab)<\/span><\/a> to enable practitioners to easily implement \u00b5P in their own PyTorch models, just as how frameworks like PyTorch, TensorFlow, and JAX have enabled us to take autograd for granted. Please note that \u00b5Transfer works for models of any size, not just those with billions of parameters. <\/p>\n\n\n\n

The journey has just begun<\/h2>\n\n\n\n

While our theory explains why models of different widths behave differently, more investigation is needed to build a theoretical understanding of the scaling of network depth and other scale dimensions. Many works have addressed the latter, such as the research on batch size by Shallue et al. (opens in new tab)<\/span><\/a>, Smith et al. (opens in new tab)<\/span><\/a>, and McCandlish et al. (opens in new tab)<\/span><\/a>, as well as research on neural language models in general by Rosenfield et al. (opens in new tab)<\/span><\/a> and Kaplan et al. (opens in new tab)<\/span><\/a> We believe \u00b5P can remove a confounding variable for such investigations.  Furthermore, recent large-scale architectures often involve scale dimensions beyond those we have talked about in our work, such as the number of experts in a mixture-of-experts system. Another high-impact domain to which \u00b5P and \u00b5Transfer have not been applied is fine tuning a pretrained model. While feature learning is crucial in that domain, the need for regularization and the finite-width effect prove to be interesting challenges. <\/p>\n\n\n\n

We firmly believe in fundamental research as a cost-effective complement to trial and error and plan to continue our work to derive more principled approaches to large-scale machine learning. To learn about our other deep learning projects or opportunities to work with us and even help us expand \u00b5P, please go to our Deep Learning Group<\/a> page.<\/p>\n","protected":false},"excerpt":{"rendered":"

Great scientific achievements cannot be made by trial and error alone. Every launch in the space program is underpinned by centuries of fundamental research in aerodynamics, propulsion, and celestial bodies. In the same way, when it comes to building large-scale AI systems, fundamental research forms the theoretical insights that drastically reduce the amount of trial […]<\/p>\n","protected":false},"author":37583,"featured_media":823717,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-author-ordering":null,"msr_hide_image_in_river":0,"footnotes":""},"categories":[1],"tags":[],"research-area":[13561,13556],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[243984],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-823648","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-research-blog","msr-research-area-algorithms","msr-research-area-artificial-intelligence","msr-locale-en_us","msr-post-option-blog-homepage-featured"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[144931],"related-projects":[804847],"related-events":[],"related-researchers":[{"type":"guest","value":"edward-hu","user_id":"822325","display_name":"Edward Hu","author_link":"Edward Hu<\/a>","is_active":true,"last_first":"Hu, Edward","people_section":0,"alias":"edward-hu"},{"type":"user_nicename","value":"Jianfeng Gao","user_id":32246,"display_name":"Jianfeng Gao","author_link":"Jianfeng Gao<\/a>","is_active":false,"last_first":"Gao, Jianfeng","people_section":0,"alias":"jfgao"}],"msr_type":"Post","featured_image_thumbnail":"\"animation","byline":"Edward Hu<\/a>, Greg Yang, and Jianfeng Gao<\/a>","formattedDate":"March 8, 2022","formattedExcerpt":"Great scientific achievements cannot be made by trial and error alone. Every launch in the space program is underpinned by centuries of fundamental research in aerodynamics, propulsion, and celestial bodies. In the same way, when it comes to building large-scale AI systems, fundamental research forms…","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/posts\/823648","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/users\/37583"}],"replies":[{"embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/comments?post=823648"}],"version-history":[{"count":39,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/posts\/823648\/revisions"}],"predecessor-version":[{"id":870630,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/posts\/823648\/revisions\/870630"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/media\/823717"}],"wp:attachment":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/media?parent=823648"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/categories?post=823648"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/tags?post=823648"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=823648"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=823648"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=823648"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=823648"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=823648"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=823648"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=823648"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=823648"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}