![\"\"](\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2020\/06\/1400x788_Xglue_NoLogo__updated.gif\")
<\/p>\n<\/p>\n
What we can teach a model to do with natural language is dictated by the availability of data. Currently, we have a lot of labeled data for very few languages, making it difficult to train models to accomplish question answering, text summarization, and other tasks in every language and ultimately limiting the amount of people who can benefit from advanced AI systems. In natural language processing (NLP), low-resource learning\u2014scenarios in which a model is trained using very little training data or data without examples of the type of data it will encounter in testing\u2014is an active challenge. Cross-lingual transfer learning is a type of low-resource learning that trains a model with data in one language, such as English, and tests the model on the same task in different languages. With cross-lingual transfer capability, we could leverage the rich resources of a few languages to build NLP services for all the languages in the world.<\/p>\n
Recently, pre-trained models such as Unicoder<\/a>, M-BERT (opens in new tab)<\/span><\/a>, and XLM (opens in new tab)<\/span><\/a> have been developed to learn multilingual representations for cross-lingual and multilingual tasks. By performing masked language model, translation language model, and other bilingual pre-training tasks on multilingual and bilingual corpora with shared vocabulary and weights for multiple languages, these models obtain surprisingly good cross-lingual capability. However, the community still lacks benchmark datasets to evaluate such capability. The Cross-Lingual Natural Language Inference (XNLI) corpus (opens in new tab)<\/span><\/a> is the most used cross-lingual benchmark for these models, but its evaluation scenario\u2014natural language inference\u2014is too simple to cover various real-world cross-lingual tasks.<\/p>\n To help the research community further advance language-agnostic models and make AI systems more inclusive, we introduce \u201cXGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation.<\/a>\u201d With its training data available only in English, XGLUE\u2019s 11 downstream tasks test a model\u2019s zero-shot cross-lingual transfer capability\u2014that is, its ability to transfer what it learned in English to the same task in other languages. In all, XGLUE covers 19 languages, including Italian, Portuguese, Russian, Swahili, and Urdu. XGLUE comprises both cross-lingual natural language understanding (NLU) tasks and<\/em> cross-lingual natural language generation (NLG) tasks and offers six new<\/em> tasks used in creating and evaluating search engine and news site scenarios, unique features that set XGLUE apart from existing NLP datasets. We also extended the universal language encoder Unicoder for a baseline, introducing two different generation pre-training tasks to accommodate NLG.<\/p>\n \t The 19 languages covered by XGLUE\u2019s 11 tasks, broken down by task. Asterisks denote the new understanding and generation tasks offered by the dataset.<\/p>\n \n\t\t\t<\/div>\n\t\t<\/div>\n\t<\/li>\n\t<\/p> \n\t<\/div>\n\t<\/p>\n We see XGLUE as an important tool in helping researchers and developers ensure that access to advanced AI systems isn\u2019t limited by the language an individual speaks. That includes those AI systems being created through AI at Scale<\/a>, the Microsoft initiative driving next-generation AI capabilities.<\/p>\n Figure 1: The new XGLUE benchmark dataset has 11 tasks, including four new NLU tasks and two new NLG tasks, denoted in the above table with an asterisk. For each task, the training set is available only in English. The third column in the table is the number of labeled instances in the training set. The fourth and fifth columns are the average numbers of labeled instances in the development sets and test sets, respectively.<\/p><\/div>\n XGLUE includes five existing NLU tasks\u2014name-entity recognition (NER), part-of-speech tagging (POS), machine reading comprehension (MLQA dataset), paraphrase classification (PAWS-X dataset), and XNLI\u2014but it\u2019s the dataset\u2019s newest tasks that give researchers an opportunity to evaluate a model\u2019s real-world potential:<\/p>\n Data for XGLUE\u2019s news classification and news title generation tasks originated from the Microsoft news website Microsoft News, while data for the remaining tasks came from the Microsoft search engine Bing. Privacy-preserving steps were taken in the collection of the data, including the removal of any data that could potentially contain personally identifiable information.<\/p>\n Collectively, these six new tasks represent much of what today\u2019s commercial search engines do and capture the user experience, providing a true test of how well a model generalizes across NLU and NLG tasks and demonstrating more concretely how a model can impact people down the line. That could mean improved direct answers to search queries, saving people the time of having to sift through pages of search results, or news organized in a way that lets people find what they want to read about more easily.<\/p>\n The XGLUE dataset (opens in new tab)<\/span><\/a> and example code for running the XGLUE baseline (opens in new tab)<\/span><\/a> are available on GitHub. Those interested in sharing their results can do so via the XGLUE leaderboard (opens in new tab)<\/span><\/a>.<\/p>\n Figure 2: Researchers used a simplified version of the universal language encoder Unicoder for an XGLUE baseline. The original Unicoder is pre-trained for cross-lingual understanding tasks using masked language model, translation language model, contrastive learning, and cross-lingual word recovery (above). Each column shows an example for each NLU task, respectively. For XGLUE, Unicoder is pre-trained by masked language model and translation language model only.<\/p><\/div>\n For our baseline, we chose Unicoder, which we introduced at the 2019 Conference on Empirical Methods in Natural Language Processing (opens in new tab)<\/span><\/a>. For cross-lingual NLU tasks, Unicoder is pre-trained using both multilingual and bilingual corpora by the following tasks (Figure 2 above): masked language model<\/em> and translation language model<\/em>, which predict each masked token or phrase based on monolingual context and bilingual context respectively; contrastive learning<\/em>, which predicts whether a bilingual word\/phrase\/sentence pair is a translation pair; and cross-lingual word recovery<\/em>, which predicts the original source sequence by its vector representations generated based on the token representations of its target translation sequence. For XGLUE, we used a simplified version of Unicoder, pre-training it by masked language model and translation language model only.<\/p>\n Figure 3: Researchers extended Unicoder for cross-lingual generation tasks using two generation tasks during pre-training: multilingual Denoising Auto-Encoding (xDAE) and multilingual Future N-gram Prediction (xFNP). In pre-training Unicoder, a text noising approach is used to corrupt a sentence, which is then used as the input of the Unicoder encoder. Then, the decoder attempts to generate the original input sequence based on its corrupted form. Researchers tried four different text noising methods and the two different text denoising methods.<\/p><\/div>Using an encoder-decoder architecture, we extend the original Unicoder to cross-lingual NLG tasks (Figure 3 above) by introducing generation tasks into the pre-training stage: multilingual Denoising Auto-Encoding (xDAE)<\/em> and multilingual Future N-gram Prediction (xFNP)<\/em>. The task of xDAE is to predict the original input text given its corrupted form, generated by text noising methods, which randomly mask, delete, or reorder words. In xDAE, the decoder generates a single token at each time step. The task of xFNP is to predict each masked span, or series of words, given the corrupted form of the input text, generated by span-based masking, which masks several successive words at once rather than a single word. In xFNP, the decoder generates multiple tokens simultaneously at each time step. Based on the already generated context and the most possible future tokens, xFNP can select the best token for the current time step. xFNP is a multilingual version of ProphetNet (opens in new tab)<\/span><\/a>. Both xFNP and ProphetNet are available on GitHub (opens in new tab)<\/span><\/a>.<\/p>\n We evaluate Unicoder and two other recent cross-lingual pre-trained models, M-BERT and XLM-RoBERTa (XLM-R) (opens in new tab)<\/span><\/a>, on XGLUE. For each task, the models are fine-tuned on its task-specific English data and then applied to all test sets, which are in a variety of languages, including English, aiming to evaluate each model\u2019s zero-shot cross-lingual transfer capability. For the cross-lingual NLU tasks, Unicoder performs slightly better than M-BERT and XLM-R because it\u2019s pre-trained using both multilingual and bilingual corpora; the other two models are pre-trained using a multilingual corpus only. For the cross-lingual NLG tasks, Unicoder performs significantly better than M-BERT and XLM-R because it introduces generation tasks into the pre-training stage; the other two models are fine-tuned on these two downstream tasks directly without pre-training.<\/p>\n We also investigated the impacts of different fine-tuning strategies:<\/p>\n With XGLUE, we seek to leverage the abundance of rich training data in English to support the development of models that can be applied to all languages\u2014models that have a truly universal language representation. Moving forward, we\u2019ll extend XGLUE to more languages and downstream tasks while continuing to push forward cross-lingual pre-trained models by exploring new model structures, introducing new pre-training tasks, using different types of data, and expanding cross-lingual pre-training to other modalities such as images and videos.<\/p>\n Acknowledgment: This research was conducted by Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, and Ming Zhou. We thank the Search Technology Center Asia NLP team, the Bing Answer team, the Bing Relevance team, the Bing Ads team, and the Microsoft News team for providing the real-world datasets.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":" What we can teach a model to do with natural language is dictated by the availability of data. Currently, we have a lot of labeled data for very few languages, making it difficult to train models to accomplish question answering, text summarization, and other tasks in every language and ultimately limiting the amount of people […]<\/p>\n","protected":false},"author":38838,"featured_media":665694,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-author-ordering":null,"msr_hide_image_in_river":0,"footnotes":""},"categories":[1],"tags":[],"research-area":[13556,13545],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[243984],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-664554","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-research-blog","msr-research-area-artificial-intelligence","msr-research-area-human-language-technologies","msr-locale-en_us","msr-post-option-blog-homepage-featured"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[199560],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[],"related-projects":[649749],"related-events":[],"related-researchers":[{"type":"user_nicename","value":"Yaobo Liang","user_id":36036,"display_name":"Yaobo Liang","author_link":"Yaobo Liang<\/a>","is_active":false,"last_first":"Liang, Yaobo","people_section":0,"alias":"yalia"},{"type":"guest","value":"daniel-campos","user_id":"664566","display_name":"Daniel Campos","author_link":"Daniel Campos<\/a>","is_active":true,"last_first":"Campos, Daniel","people_section":0,"alias":"daniel-campos"}],"msr_type":"Post","featured_image_thumbnail":"\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t style=\"margin: 0in; font-family: Calibri; font-size: 11.0pt;\">\t
\n\t\t\t\n\t\t\t\t<\/p>\n![\"\"](\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2020\/06\/Updated-XGLUE-languages-table.jpg\")
<\/p>\n![\"\"](\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2020\/06\/Figure-1-_-XGLUE-1024x348.png\")
New tasks\u2014including NLG tasks\u2014from real-world scenarios<\/h3>\n
\n
\n
![\"A](\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2020\/06\/Unicoder-Updated-Fig-1-1024x508.jpg\")
Cross-lingual pre-training in Unicoder<\/h3>\n
![\"A](\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2020\/06\/Unicoder-Updated-Fig-2-1024x445.jpg\")
<\/h3>\n
Evaluation results on XGLUE<\/h3>\n
\n
Looking forward<\/h3>\n
![\"\"](\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2020\/06\/XGLUE-homepage-feat-image-960x540.png\")
","byline":"Nan Duan, Yaobo Liang<\/a>, and Daniel Campos<\/a>","formattedDate":"June 11, 2020","formattedExcerpt":"What we can teach a model to do with natural language is dictated by the availability of data. Currently, we have a lot of labeled data for very few languages, making it difficult to train models to accomplish question answering, text summarization, and other tasks…","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/posts\/664554","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/users\/38838"}],"replies":[{"embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/comments?post=664554"}],"version-history":[{"count":10,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/posts\/664554\/revisions"}],"predecessor-version":[{"id":665733,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/posts\/664554\/revisions\/665733"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/media\/665694"}],"wp:attachment":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/media?parent=664554"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/categories?post=664554"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/tags?post=664554"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=664554"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=664554"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=664554"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=664554"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=664554"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=664554"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=664554"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=664554"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}