{"id":1045257,"date":"2024-06-27T20:19:16","date_gmt":"2024-06-28T03:19:16","guid":{"rendered":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/?post_type=msr-project&#038;p=1045257"},"modified":"2024-07-18T15:13:34","modified_gmt":"2024-07-18T22:13:34","slug":"emoctrl-tts","status":"publish","type":"msr-project","link":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/project\/emoctrl-tts\/","title":{"rendered":"EmoCtrl-TTS"},"content":{"rendered":"<section class=\"mb-3 moray-highlight\">\n\t<div class=\"card-img-overlay mx-lg-0\">\n\t\t<div class=\"card-background  has-background-catalina-blue card-background--full-bleed\">\n\t\t\t\t\t<\/div>\n\t\t<!-- Foreground -->\n\t\t<div class=\"card-foreground d-flex mt-md-n5 my-lg-5 px-g px-lg-0\">\n\t\t\t<!-- Container -->\n\t\t\t<div class=\"container d-flex mt-md-n5 my-lg-5 \">\n\t\t\t\t<!-- Card wrapper -->\n\t\t\t\t<div class=\"w-100 w-lg-col-5\">\n\t\t\t\t\t<!-- Card -->\n\t\t\t\t\t<div class=\"card material-md-card py-5 px-md-5\">\n\t\t\t\t\t\t<div class=\"card-body \">\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\n<h1 class=\"wp-block-heading\" id=\"emoctrl-tts\">EmoCtrl-TTS<\/h1>\n\n\n\n<p>Controlling Time-Varying Emotional States of Flow-Matching-Based Zero-Shot Text-to-Speech<\/p>\n\n\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t<\/div>\n<\/section>\n\n\n\n\n\n<p><strong>EmoCtrl-TTS<\/strong> is an emotion-controllable zero-shot TTS that can generate <strong>highly emotional speech with non-verbal vocalizations such as laughter and crying<\/strong> for any speaker. EmoCtrl-TTS is purely a research project. Currently, we have no plans to incorporate EmoCtrl-TTS into a product or expand access to the public.<\/p>\n\n\n\n<div class=\"wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex\">\n<div class=\"wp-block-button\"><a data-bi-type=\"button\" class=\"wp-block-button__link wp-element-button\" href=\"https:\/\/arxiv.org\/abs\/2407.12229\" target=\"_blank\" rel=\"noreferrer noopener\">Read the paper<\/a><\/div>\n<\/div>\n\n\n\n<h2 class=\"wp-block-heading has-text-align-center is-style-default\" id=\"controlling-time-varying-emotional-states-of-zero-shot-text-to-speech\">Controlling time-varying emotional states of zero-shot text-to-speech<\/h2>\n\n\n\n<p>EmoCtrl-TTS utilizes embeddings that represent emotion and non-verbal vocalizations to condition the flow-matching-based zero-shot TTS. In order to generate high-quality emotional speech, EmoCtrl-TTS is trained with over 27,000 hours of expressive data, curated using pseudo-labeling.<\/p>\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-9d6595d7 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-vertically-aligned-center is-layout-flow wp-block-column-is-layout-flow\" style=\"flex-basis:66.66%\">\n<figure class=\"wp-block-image aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"276\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/Overview-v3-1024x276.png\" alt=\"Overview\" class=\"wp-image-1050993\" srcset=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/Overview-v3-1024x276.png 1024w, https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/Overview-v3-300x81.png 300w, https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/Overview-v3-768x207.png 768w, https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/Overview-v3-1536x414.png 1536w, https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/Overview-v3-2048x551.png 2048w, https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/Overview-v3-240x65.png 240w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\" style=\"flex-basis:33.33%\">\n<p><strong>EmoCtrl-TTS can generate a voice of any speaker with non-verbal vocalizations like laughter and crying.<\/strong><\/p>\n\n\n\n<p class=\"has-text-align-center\">Generated speech samples <\/p>\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-9d6595d7 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<div style=\"text-align: center\">\n<audio controls style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/R8005_M8008_N_SPK8042_632.04_636.5144316426334_emoctrl.wav\"><\/audio>\n<\/div>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<div style=\"text-align: center\">\n<audio controls style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/M2_sad_regular_39_emoctrl.wav\"><\/audio>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n<p><strong>EmoCtrl-TTS is specifically designed to capture the time-varying emotional states found in the audio prompt.<\/strong><\/p>\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-9d6595d7 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<p class=\"has-text-align-center\">Audio prompt (Angry \u2192 Calm)<\/p>\n\n\n\n<div style=\"text-align: center\">\n<audio controls style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/M_spk_01-angrycalm_reference.wav\"><\/audio>\n<\/div>\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-9d6595d7 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<p class=\"has-text-align-center\">Generate speech by Voicebox (prior work)<\/p>\n\n\n\n<div style=\"text-align: center\">\n<audio controls style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/M_spk_01-angrycalm_baseline_tts.wav\"><\/audio>\n<\/div>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<p class=\"has-text-align-center\">Generated speech by EmoCtrl-TTS (our work)<\/p>\n\n\n\n<div style=\"text-align: center\">\n<audio controls style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/M_spk_01-angrycalm_emoctrl.wav\"><\/audio>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-9d6595d7 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\"><\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\"><\/div>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n<h2 class=\"wp-block-heading has-text-align-center is-style-default\" id=\"audio-samples\">Audio samples<\/h2>\n\n\n\n<p>Below, we included audio samples demonstrating how EmoCtrl-TTS performs. The speech samples were taken from JVNV dataset, DiariST-AliMeeting dataset, and RAVDESS dataset. The speech samples below are provided for the sole purpose of illustrating EmoCtrl-TTS.<\/p>\n\n\n\n<div style=\"margin-bottom: 50px\">\n    <h3 class=\"wp-block-heading is-style-l\" id=\"specifying-the-pronunciation-without-model-re-training\">Capturing the time-varying emotional states<\/h3>\n <p style=\"text-align: left\">EmoCtrl-TTS can generate a speech that closely mimics the time-varying emotional states found in the audio prompt. In these demo samples, the audio prompt is created by concatenating two audio samples from RAVDESS data set. The text prompt is <i>&#8220;dogs are sitting by the door dogs are sitting by the door&#8221;<\/i> for all generated speech samples. <\/p>\n     <div style=\"border-bottom: 2px solid black;margin-bottom: 2px\"><\/div>\n     <div style=\"background-color: #E6E6FA;padding: 20px;border-radius: 5px;max-width: 80%;margin: 20px auto\">\n         <table style=\"width: 100%;border-collapse: collapse;border: none\">\n\n                <thead>\n                <tr style=\"border-bottom: 2px solid black\">\n                    <th style=\"text-align: center;padding: 8px\" rowspan=\"2\">Emotion<\/th>\n                    <th style=\"text-align: center;padding: 8px\" rowspan=\"2\">Audio prompt<\/th>\n                    <th style=\"text-align: center;padding: 8px\" colspan=\"3\">Generated audio<\/th>\n                <\/tr>\n                <tr style=\"border-bottom: 2px solid black\">\n                    <th style=\"text-align: center;padding: 8px\">Voicebox<\/th>\n                    <th style=\"text-align: center;padding: 8px\">ELaTE<\/th>\n                    <th style=\"text-align: center;padding: 8px\">EmoCtrl-TTS<\/th>\n                <\/tr>\n                <\/thead>\n             <tbody>\n\n             <!-- happy --> \n             <tr>\n                 <!-- F1 --> \n                 <td style=\"text-align: left;padding: 8px;border-bottom: 1px solid #ccc\" rowspan=\"2\">\n\nAngry &rarr; Calm<\/td>\n\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/M_spk_01-angrycalm_reference.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/M_spk_01-angrycalm_baseline_tts.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/M_spk_01-angrycalm_elate.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/M_spk_01-angrycalm_emoctrl.wav\"><\/audio>\n                 <\/td>\n             <\/tr>\n                 <tr><td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/F_spk_02-angrycalm_reference.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/F_spk_02-angrycalm_baseline_tts.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/F_spk_02-angrycalm_elate.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/F_spk_02-angrycalm_emoctrl.wav\"><\/audio>\n                 <\/td>\n             <\/tr>\n\n             <tr>\n                 <!-- F1 --> \n                 <td style=\"text-align: left;padding: 8px;border-bottom: 1px solid #ccc\" rowspan=\"2\">\n\nSad &rarr; Surprised<\/td>\n\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/M_spk_01-sadsurprised_reference.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/M_spk_01-sadsurprised_baseline_tts.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/M_spk_01-sadsurprised_elate.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/M_spk_01-sadsurprised_emoctrl.wav\"><\/audio>\n                 <\/td>\n             <\/tr>\n                 <tr><td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/F_spk_20-sadsurprised_reference.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/F_spk_20-sadsurprised_baseline_tts.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/F_spk_20-sadsurprised_elate.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/F_spk_20-sadsurprised_emoctrl.wav\"><\/audio>\n                 <\/td>\n             <\/tr>\n\n             <tr>\n                 <td style=\"text-align: left;padding: 8px;border-bottom: 1px solid #ccc\" rowspan=\"2\">\n\n\n\nHappy &rarr; Disgusted<\/td>\n\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/M_spk_03-happydisgusted_reference.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/M_spk_03-happydisgusted_baseline_tts.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/M_spk_03-happydisgusted_elate.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/M_spk_03-happydisgusted_emoctrl.wav\"><\/audio>\n                 <\/td>\n             <\/tr>\n                 <tr><td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/F_spk_06-happydisgusted_reference.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/F_spk_06-happydisgusted_baseline_tts.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/F_spk_06-happydisgusted_elate.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/F_spk_06-happydisgusted_emoctrl.wav\"><\/audio>\n                 <\/td>\n             <\/tr>\n\n             <tr>\n                 <td style=\"text-align: left;padding: 8px;border-bottom: 1px solid #ccc\" rowspan=\"2\">\n\nCalm &rarr; Fearful<\/td>\n\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/M_spk_11-calmfearful_reference.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/M_spk_11-calmfearful_baseline_tts.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/M_spk_11-calmfearful_elate.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/M_spk_11-calmfearful_emoctrl.wav\"><\/audio>\n                 <\/td>\n             <\/tr>\n                 <tr><td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/F_spk_22-calmfearful_reference.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/F_spk_22-calmfearful_baseline_tts.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/F_spk_22-calmfearful_elate.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/F_spk_22-calmfearful_emoctrl.wav\"><\/audio>\n                 <\/td>\n             <\/tr>\n\n             <\/tbody>\n         <\/table>\n     <\/div>\n <\/div>\n\n\n\n<div style=\"margin-bottom: 50px\">\n    <h3 class=\"wp-block-heading is-style-l\" id=\"specifying-the-pronunciation-without-model-re-training\">Generating non-verbal vocalization<\/h3>\n <p style=\"text-align: left\">EmoCtrl-TTS can generate non-verbal vocalizations, such as laughter and crying, that closely match the audio prompt.<\/p>\n     <div style=\"border-bottom: 2px solid black;margin-bottom: 2px\"><\/div>\n     <h3 style=\"text-align: center\">          <\/h3>\n     <h6 style=\"text-align: center\"> Laughing speech generation <\/h6>\n     <p style=\"text-align: center\"> (Audio prompt from AliMeeting-DiariST dataset; real conversational speech in Chinese) <\/p>\n     <div style=\"background-color: #E6E6FA;padding: 20px;border-radius: 5px;max-width: 80%;margin: 20px auto\">\n         <table style=\"width: 100%;border-collapse: collapse;border: none\">\n\n                <thead>\n                <tr style=\"border-bottom: 2px solid black\">\n                    <th style=\"text-align: center;padding: 8px\" rowspan=\"2\">Audio prompt (Chinese)<\/th>\n                    <th style=\"text-align: center;padding: 8px\" rowspan=\"2\">Text prompt (English)<\/th>\n                    <th style=\"text-align: center;padding: 8px\" colspan=\"3\">Generated audio<\/th>\n                <\/tr>\n                <tr style=\"border-bottom: 2px solid black\">\n                    <th style=\"text-align: center;padding: 8px\">Voicebox<\/th>\n                    <th style=\"text-align: center;padding: 8px\">ELaTE<\/th>\n                    <th style=\"text-align: center;padding: 8px\">EmoCtrl-TTS<\/th>\n                <\/tr>\n                <\/thead>\n\n             <tbody>\n\n             <tr>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/R8004_M8006_N_SPK8033_756.21_765.01_ref.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px\">Ah, look, right, isn&#8217;t it? At a glance, oh, yes, then maybe play for a while. Oh, maybe we&#8217;ll be fine.<\/td>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/R8004_M8006_N_SPK8033_756.21_765.01_baseline_tts.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/R8004_M8006_N_SPK8033_756.21_765.01_elate.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/R8004_M8006_N_SPK8033_756.21_765.01_emoctrl.wav\"><\/audio>\n                 <\/td>\n             <\/tr>\n                 \n             <tr>\n\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/R8005_M8008_N_SPK8042_632.04_636.5144316426334_ref.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px\">You remind me of the kitchen knives sold in the morning market.<\/td>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/R8005_M8008_N_SPK8042_632.04_636.5144316426334_baseline_tts.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/R8005_M8008_N_SPK8042_632.04_636.5144316426334_elate.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/R8005_M8008_N_SPK8042_632.04_636.5144316426334_emoctrl.wav\"><\/audio>\n                 <\/td>\n             <\/tr>\n\n\n             <tr>\n\n                 \n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/R8004_M8005_N_SPK8019_187.42_193.42_ref.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px\">But I think buying these financial products won&#8217;t be fooled.<\/td>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/R8004_M8005_N_SPK8019_187.42_193.42_baseline_tts.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/R8004_M8005_N_SPK8019_187.42_193.42_elate.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/R8004_M8005_N_SPK8019_187.42_193.42_emoctrl.wav\"><\/audio>\n                 <\/td>\n             <\/tr>\n                \n\n             <tr>\n\n                 \n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/R8004_M8005_N_SPK8017_1240.38_1246.2828734497934_ref.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px\">But don&#8217;t you think after seeing that number you feel very panicked and very uncomfortable inside?<\/td>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/R8004_M8005_N_SPK8017_1240.38_1246.2828734497934_baseline_tts.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/R8004_M8005_N_SPK8017_1240.38_1246.2828734497934_elate.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/R8004_M8005_N_SPK8017_1240.38_1246.2828734497934_emoctrl.wav\"><\/audio>\n                 <\/td>\n             <\/tr>\n                \n\n             <tr>\n\n                 \n                <td style=\"text-align: center;padding: 8px\">\n                    <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/R8005_M8009_N_SPK8045_1222.5892436656836_1228.5179354434672_ref.wav\"><\/audio>\n                <\/td>\n                 <td style=\"text-align: center;padding: 8px\">You take a look at your share first.<\/td>\n                <td style=\"text-align: center;padding: 8px\">\n                    <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/R8005_M8009_N_SPK8045_1222.5892436656836_1228.5179354434672_baseline_tts.wav\"><\/audio>\n                <\/td>\n                <td style=\"text-align: center;padding: 8px\">\n                    <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/R8005_M8009_N_SPK8045_1222.5892436656836_1228.5179354434672_elate.wav\"><\/audio>\n                <\/td>\n                <td style=\"text-align: center;padding: 8px\">\n                    <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/R8005_M8009_N_SPK8045_1222.5892436656836_1228.5179354434672_emoctrl.wav\"><\/audio>\n                <\/td>\n            <\/tr>\n\n             <\/tbody>\n         <\/table>\n     <\/div>\n\n\n     <h6 style=\"text-align: center\">Crying speech generation<\/h6>\n     <p style=\"text-align: center\">(Audio prompt from JVNV dataset; staged speech in Japanese) <\/p>\n     <div style=\"background-color: #E6E6FA;padding: 20px;border-radius: 5px;max-width: 80%;margin: 20px auto\">\n         <table style=\"width: 100%;border-collapse: collapse;border: none\">\n\n\n                <thead>\n                <tr style=\"border-bottom: 2px solid black\">\n                    <th style=\"text-align: center;padding: 8px\" rowspan=\"2\">Audio prompt (Japanese)<\/th>\n                    <th style=\"text-align: center;padding: 8px\" rowspan=\"2\">Text prompt (English)<\/th>\n                    <th style=\"text-align: center;padding: 8px\" colspan=\"3\">Generated audio<\/th>\n                <\/tr>\n                <tr style=\"border-bottom: 2px solid black\">\n                    <th style=\"text-align: center;padding: 8px\">Voicebox<\/th>\n                    <th style=\"text-align: center;padding: 8px\">ELaTE<\/th>\n                    <th style=\"text-align: center;padding: 8px\">EmoCtrl-TTS<\/th>\n                <\/tr>\n                <\/thead>\n             <tbody>\n\n             <tr>\n                 \n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/M2_sad_regular_39_reference.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px\">Our team suffered a huge defeat today. I deeply regret holding everyone back.<\/td>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/M2_sad_regular_39_baseline_tts.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/M2_sad_regular_39_elate.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/M2_sad_regular_39_emoctrl.wav\"><\/audio>\n                 <\/td>\n             <\/tr>\n             <tr>\n                 \n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/F2_sad_free_05_reference.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px\">Ever since she became depressed, every day has been gloomy and painful. I want to help, but I don&#8217;t know what to do.<\/td>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/F2_sad_free_05_baseline_tts.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/F2_sad_free_05_elate.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/F2_sad_free_05_emoctrl.wav\"><\/audio>\n                 <\/td>\n             <\/tr>\n\n\n             <tr>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/M1_sad_regular_10_reference.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px\">Ah, last night, I got into a car accident and the other person passed away. It&#8217;s so painful to be alive, I can&#8217;t help it.<\/td>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/M1_sad_regular_10_baseline_tts.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/M1_sad_regular_10_elate.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/M1_sad_regular_10_emoctrl.wav\"><\/audio>\n                 <\/td>\n             <\/tr>\n             \n             <tr>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/F1_sad_regular_27_reference.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px\">I ruined an important friendship. Why did I do such a thing?<\/td>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/F1_sad_regular_27_baseline_tts.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/F1_sad_regular_27_elate.wav\"><\/audio>\n                 <\/td>\n                 <td style=\"text-align: center;padding: 8px\">\n                     <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/F1_sad_regular_27_emoctrl.wav\"><\/audio>\n                 <\/td>\n             <\/tr>\n\n\n\n             <tr>\n                 \n                <td style=\"text-align: center;padding: 8px\">\n                    <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/F2_sad_free_02_reference.wav\"><\/audio>\n                <\/td>\n                 <td style=\"text-align: center;padding: 8px\">Ugh, my brother drowned in the sea yesterday. I cried all night in grief.<\/td>\n                <td style=\"text-align: center;padding: 8px\">\n                    <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/F2_sad_free_02_baseline_tts.wav\"><\/audio>\n                <\/td>\n                <td style=\"text-align: center;padding: 8px\">\n                    <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/F2_sad_free_02_elate.wav\"><\/audio>\n                <\/td>\n                <td style=\"text-align: center;padding: 8px\">\n                    <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/F2_sad_free_02_emoctrl.wav\"><\/audio>\n                <\/td>\n            <\/tr>\n\n             <\/tbody>\n         <\/table>\n     <\/div>\n <\/div>\n\n\n\n<div style=\"margin-bottom: 50px\">\n       <h3 class=\"wp-block-heading is-style-l\" id=\"specifying-the-pronunciation-without-model-re-training\">Emotional speech-to-speech translation<\/h3>\n    <p style=\"text-align: left\">EmoCtrl-TTS can be applied to speech-to-speech translation, transferring not only the voice characteristic but also the precise nuance of the source audio. The source audios were sampled from the JNVN dataset, which is a Japanese staged emotional speech corpus.<\/p>\n        <div style=\"border-bottom: 2px solid black;margin-bottom: 2px\"><\/div>\n        <div style=\"background-color: #E6E6FA;padding: 20px;border-radius: 5px;max-width: 80%;margin: 20px auto\">\n            <table style=\"width: 100%;border-collapse: collapse;border: none\">\n                <thead>\n                <tr style=\"border-bottom: 2px solid black\">\n                    <th style=\"text-align: center;padding: 8px\" rowspan=\"2\">Emotion<\/th>\n                    <th style=\"text-align: center;padding: 8px\" rowspan=\"2\">Source audio (Japanese)<\/th>\n                    <th style=\"text-align: center;padding: 8px\" colspan=\"3\">Translated audio (English)<\/th>\n                <\/tr>\n                <tr style=\"border-bottom: 2px solid black\">\n                    <th style=\"text-align: center;padding: 8px\">SeamlessExpressive<sup>(*)<\/sup><\/th>\n                    <th style=\"text-align: center;padding: 8px\">Voicebox<sup>(**)<\/sup><\/th>\n                    <th style=\"text-align: center;padding: 8px\">ELaTE<sup>(**)<\/sup><\/th>\n                    <th style=\"text-align: center;padding: 8px\">EmoCtrl-TTS<sup>(**)<\/sup><\/th>\n                <\/tr>\n                <\/thead>\n                <tbody>\n\n                <!-- happy --> \n                <tr>\n                    <!-- F1 --> \n                    <td style=\"text-align: left;padding: 8px;border-bottom: 1px solid #ccc\" rowspan=\"2\">\n\nHappy<\/td>\n\n                    <td style=\"text-align: center;padding: 8px\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_F1_happy_reference_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_F1_happy_Seamless_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_F1_happy_paper_B5_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_F1_happy_ELaTE_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_F1_happy_EmoCtrl_v2.wav\"><\/audio>\n                    <\/td>\n                <\/tr>\n                    <!-- M1 --> \n                    <tr><td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_M1_happy_reference_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_M1_happy_Seamless_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_M1_happy_paper_B5_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_M1_happy_ELaTE_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_M1_happy_EmoCtrl_v2.wav\"><\/audio>\n                    <\/td>\n                <\/tr>\n\n\n                <!-- sad --> \n                <tr>\n                    <!-- F1 --> \n                    <td style=\"text-align: left;padding: 8px;border-bottom: 1px solid #ccc\" rowspan=\"2\">\n\nSad<\/td>\n\n                    <td style=\"text-align: center;padding: 8px\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_F1_sad_reference_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_F1_sad_Seamless_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_F1_sad_paper_B5_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_F1_sad_ELaTE_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_F1_sad_EmoCtrl_v2.wav\"><\/audio>\n                    <\/td>\n                <\/tr>\n                    <!-- M1 --> \n                    <tr><td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_M1_sad_reference_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_M1_sad_Seamless_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_M1_sad_paper_B5_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_M1_sad_ELaTE_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_M1_sad_EmoCtrl_v2.wav\"><\/audio>\n                    <\/td>\n                <\/tr>\n\n\n                <!-- angry --> \n                <tr>\n                    <!-- F1 --> \n                    <td style=\"text-align: left;padding: 8px;border-bottom: 1px solid #ccc\" rowspan=\"2\">\n\nAngry<\/td>\n\n                    <td style=\"text-align: center;padding: 8px\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_F1_angry_reference_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_F1_angry_Seamless_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_F1_angry_paper_B5_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_F1_angry_ELaTE_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_F1_angry_EmoCtrl_v2.wav\"><\/audio>\n                    <\/td>\n                <\/tr>\n                    <!-- M1 --> \n                    <tr><td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_M1_angry_reference_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_M1_angry_Seamless_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_M1_angry_paper_B5_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_M1_angry_ELaTE_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_M1_angry_EmoCtrl_v2.wav\"><\/audio>\n                    <\/td>\n                <\/tr>\n\n\n                <!-- surprised --> \n                <tr>\n                    <!-- F1 --> \n                    <td style=\"text-align: left;padding: 8px;border-bottom: 1px solid #ccc\" rowspan=\"2\">\n\nSurprised<\/td>\n\n                    <td style=\"text-align: center;padding: 8px\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_F1_surprised_reference_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_F1_surprised_Seamless_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_F1_surprised_paper_B5_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_F1_surprised_ELaTE_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_F1_surprised_EmoCtrl_v2.wav\"><\/audio>\n                    <\/td>\n                <\/tr>\n                    <!-- M1 --> \n                    <tr><td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_M1_surprised_reference_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_M1_surprised_Seamless_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_M1_surprised_paper_B5_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_M1_surprised_ELaTE_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_M1_surprised_EmoCtrl_v2.wav\"><\/audio>\n                    <\/td>\n                <\/tr>\n\n\n                <!-- disgusted --> \n                <tr>\n                    <!-- F1 --> \n                    <td style=\"text-align: left;padding: 8px;border-bottom: 1px solid #ccc\" rowspan=\"2\">\n\nDisgusted<\/td>\n\n                    <td style=\"text-align: center;padding: 8px\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_F1_disgusted_reference_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_F1_disgusted_Seamless_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_F1_disgusted_paper_B5_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_F1_disgusted_ELaTE_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_F1_disgusted_EmoCtrl_v2.wav\"><\/audio>\n                    <\/td>\n                <\/tr>\n                    <!-- M1 --> \n                    <tr><td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_M1_disgusted_reference_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_M1_disgusted_Seamless_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_M1_disgusted_paper_B5_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_M1_disgusted_ELaTE_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_M1_disgusted_EmoCtrl_v2.wav\"><\/audio>\n                    <\/td>\n                <\/tr>\n\n\n                <!-- fearful --> \n                <tr>\n                    <!-- F1 --> \n                    <td style=\"text-align: left;padding: 8px;border-bottom: 1px solid #ccc\" rowspan=\"2\">\n\nFearful<\/td>\n\n                    <td style=\"text-align: center;padding: 8px\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_F1_fearful_reference_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_F1_fearful_Seamless_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_F1_fearful_paper_B5_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_F1_fearful_ELaTE_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_F1_fearful_EmoCtrl_v2.wav\"><\/audio>\n                    <\/td>\n                <\/tr>\n                    <!-- M2 --> \n                    <tr><td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_M2_fearful_reference_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_M2_fearful_Seamless_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_M2_fearful_paper_B5_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_M2_fearful_ELaTE_v2.wav\"><\/audio>\n                    <\/td>\n                    <td style=\"text-align: center;padding: 8px;border-bottom: 1px solid #ccc\">\n                        <audio controls=\"\" style=\"width: 150px\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2024\/06\/JVNV_M2_fearful_EmoCtrl_v2.wav\"><\/audio>\n                    <\/td>\n                <\/tr>\n\n\n\n\n                <\/tbody>\n            <\/table>\n        <\/div>\n    <\/div>\n\n\n\n<p><em><sup>(*) We used Seamless Expressive for a pure research purpose. Seamless Expressive was used based on the Seamless Licensing Agreement. Copyright \u00a9 Meta Platforms, Inc. All Rights Reserved.<\/sup><\/em><br><em><sup>(**) We used Whisper to transcribe the speech, and then applied GPT-4 to translate the transcription to English. <\/sup><\/em><\/p>\n\n\n\n<h2 class=\"wp-block-heading has-text-align-center is-style-default\" id=\"ethics-statement-1\">Ethics statement<\/h2>\n\n\n\n<p>EmoCtrl-TTS is purely a research project. Currently, we have no plans to incorporate EmoCtrl-TTS into a product or expand access to the public. EmoCtrl-TTS could synthesize speech that maintains speaker identity and could be used for educational learning, entertainment, journalistic, self-authored content, accessibility features, interactive voice response systems, translation, chatbot, and so on. While EmoCtrl-TTS can speak in a voice like the voice talent, the similarity, and naturalness depend on the length and quality of the speech prompt, the background noise, as well as other factors. It may carry potential risks in the misuse of the model, such as spoofing voice identification or impersonating a specific speaker. We conducted the experiments under the assumption that the user agrees to be the target speaker in speech synthesis. If the model is generalized to unseen speakers in the real world, it should include a protocol to ensure that the speaker approves the use of their voice and a synthesized speech detection model. If you suspect that EmoCtrl-TTS is being used in a manner that is abusive or illegal or infringes on your rights or the rights of other people, you can report it at the Report Abuse Portal.<\/p>\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-9d6595d7 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\"><\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\"><\/div>\n<\/div>\n\n\n\n\n\n<p><\/p>\n\n\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-9d6595d7 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\"><\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\"><\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>Controlling Time-Varying Emotional States of Flow-Matching-Based Zero-Shot Text-to-Speech EmoCtrl-TTS is an emotion-controllable zero-shot TTS that can generate highly emotional speech with non-verbal vocalizations such as laughter and crying for any speaker. EmoCtrl-TTS is purely a research project. Currently, we have no plans to incorporate EmoCtrl-TTS into a product or expand access to the public. EmoCtrl-TTS [&hellip;]<\/p>\n","protected":false},"featured_media":0,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","footnotes":""},"research-area":[13556,13545],"msr-locale":[268875],"msr-impact-theme":[],"msr-pillar":[],"class_list":["post-1045257","msr-project","type-msr-project","status-publish","hentry","msr-research-area-artificial-intelligence","msr-research-area-human-language-technologies","msr-locale-en_us","msr-archive-status-active"],"msr_project_start":"","related-publications":[],"related-downloads":[],"related-videos":[],"related-groups":[],"related-events":[],"related-opportunities":[],"related-posts":[],"related-articles":[],"tab-content":[],"slides":[],"related-researchers":[{"type":"user_nicename","display_name":"Xiaofei Wang","user_id":38658,"people_section":"Related people","alias":"xiaofewa"},{"type":"user_nicename","display_name":"Manthan Thakker","user_id":39627,"people_section":"Related people","alias":"mathakke"}],"msr_research_lab":[199565],"msr_impact_theme":[],"_links":{"self":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/1045257","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-project"}],"about":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-project"}],"version-history":[{"count":95,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/1045257\/revisions"}],"predecessor-version":[{"id":1058450,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/1045257\/revisions\/1058450"}],"wp:attachment":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/media?parent=1045257"}],"wp:term":[{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=1045257"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=1045257"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=1045257"},{"taxonomy":"msr-pillar","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-pillar?post=1045257"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}