{"id":742696,"date":"2021-05-03T11:51:37","date_gmt":"2021-05-03T18:51:37","guid":{"rendered":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/?p=742696"},"modified":"2023-03-14T15:41:08","modified_gmt":"2023-03-14T22:41:08","slug":"conversations-with-data-advancing-the-state-of-the-art-in-language-driven-data-exploration","status":"publish","type":"post","link":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/blog\/conversations-with-data-advancing-the-state-of-the-art-in-language-driven-data-exploration\/","title":{"rendered":"Conversations with data: Advancing the state of the art in language-driven data exploration"},"content":{"rendered":"\n
\"Flow<\/figure>\n\n\n\n

One key aspiration of AI is to develop natural and effective task-oriented conversational systems. Task-oriented conversational systems use a natural language interface to collaborate with and support people in accomplishing specific goals and activities. They go beyond chitchat conversation. For example, as personal digital assistants, they ease the stress of trip planning or reduce the expertise required to generate a sales report from a database. While natural language understanding (NLU) technology and research have achieved remarkable recent progress, task-oriented assistance requires tackling additional challenges in practical NLU.<\/p>\n\n\n\n

Consider a prime application of task-oriented conversations: language-driven data<\/em> exploration<\/em>. Data scientists, analysts, and information workers routinely spend more than half of their time exploring, visualizing, and reformatting datasets, according to Anaconda\u2019s \u201cThe State of Data Science 2020 (opens in new tab)<\/span><\/a>.\u201d Not only time-consuming, this process is also error prone and typically requires data science programming skills, such as knowledge in Python, R, or SQL. Augmenting interactive data science environments like Microsoft Excel <\/a>or Jupyter (opens in new tab)<\/span><\/a> with language-driven assistance would not only save time but also democratize data exploration. For instance, an analyst could ask a system in natural language to plot last month\u2019s sales metrics from her database rather than program a filtered visualization in SQL+R. Importantly, such a system should still allow analysts to inspect and edit program snippets after assisting with the most laborious parts of exploratory data science. This transparency and ability to edit will empower analysts and allow them to have confidence in the outcome of the work.<\/p>\n\n\n\n

Data exploration highlights a core NLU challenge that plagues all task-oriented conversational systems. Understanding people\u2019s language\u2014and thus, their task intent<\/em>\u2014must be grounded<\/em>. That is, what has been said must be interpreted relative to its context. Task-oriented systems deal with two kinds of relevant context. First, every task acts upon a structured ontology such as a database, a spreadsheet, or an API. The ontology provides data context, which influences language understanding. For example, the analyst question \u201cWhich departments have unfinished projects?<\/em>\u201d refers to \u201cdepartments<\/em>\u201d and \u201cprojects<\/em>\u201d in her database and \u201cunfinished<\/em>\u201d likely refers to a project\u2019s status column in that database, all of which the system must emit as column references in the desired SQL program. Second, conversational systems must consider multi-turn dynamics of an interaction, which create conversation context.<\/em> For example, the analyst might follow up her exploration with \u201cWhat is their total budget,<\/em>\u201d implicitly referring to \u201cunfinished projects\u201d from the previous turn.<\/p>\n\n\n\n

\n\t
\n\t\t
\n\t\t\t\t\t\tPublication<\/span>\n\t\t\tSCoRe: Pre-Training for Context Representation in Conversational Semantic Parsing<\/span> <\/span><\/a>\t\t\t\t\t<\/div>\n\t<\/article>\n<\/div>\n\n\n\n

Recently, we\u2019ve made several fundamental contributions<\/a> to these challenges. In the 2021 International Conference on Learning Representations (ICLR)<\/a> publication \u201cSCoRe: Pre-Training for Context Representation in Conversational Semantic Parsing,\u201d<\/a> we introduce SCoRe, a task-oriented conversational system with multiple applications. SCoRe achieves new state-of-the-art performance in interactive data exploration (on SParC (opens in new tab)<\/span><\/a> and CoSQL (opens in new tab)<\/span><\/a> benchmarks) and task-oriented dialogue (MultiWOZ (opens in new tab)<\/span><\/a>), improving upon previous best techniques by up to 12 percent. SCoRe addresses the conversation context challenge through its task-oriented pretraining methodology, which learns language representations that link multiple conversation turns. To address the data context challenge, SCoRe builds upon our previous work in RAT-SQL<\/a> and StruG<\/a>. These two publications introduce a unified framework for language understanding in the context of a structured database. It has since been leveraged in numerous applications in addition to SCoRe. We\u2019re presenting SCoRe at ICLR on Monday, May 3, from 5 PM to 7 PM Pacific Time and 7 PM to 9 PM Pacific Time. The SCoRe code will be published on GitHub (opens in new tab)<\/span><\/a>; please follow the repository for updates.<\/p>\n\n\n\n

\"Two<\/a>
Figure 1: Achieving natural and effective task-oriented conversational systems requires the ability to interpret natural language based on data context\u2014how a query relates to the ontology over which it is being made\u2014and conversation context\u2014how a query relates to previous conversational turns. Above are examples of conversational tasks from the text-to-SQL dataset SParC and the dialogue modeling dataset MultiWOZ, which exhibit different forms of such context for the system to learn.<\/figcaption><\/figure>\n\n\n\n

Data context representation<\/h2>\n\n\n\n

The grounding challenges associated with data context and conversation context are distinct yet interconnected, and progress on both is critical to build effective task-oriented conversational systems. Here, we first address data context grounding by focusing on a single-turn version of the data exploration problem known as database question answering (DBQA).<\/em> As you\u2019ll see, techniques developed for DBQA facilitate data context grounding in broader applications of task-oriented conversational systems.<\/p>\n\n\n\n

RAT-SQL: Joint representation of question and data context<\/h3>\n\n\n\n

When an analyst asks a question over her database, that question and its associated data context\u2014the database schema<\/em>\u2014are both embedded into distributed intent representations<\/em> by a neural encoder network. The key to addressing the data context challenge is jointly <\/em>contextualizing intent representations\u2014the question and the schema provide important context to each other. For example, in Figure 1, \u201cFind the names of the top 3 highest sales books\u201d<\/em> refers to the title <\/code>column even though the question doesn\u2019t mention \u201ctitles<\/em>.\u201d From question language alone, author.name<\/code> could superficially seem a better match.<\/p>\n\n\n\n

Transformers (opens in new tab)<\/span><\/a> are the most effective approach for contextualized representation learning in modern NLU. They\u2019re based on self-attention<\/em>, which, in one interpretation, learns latent relations between the inputs\u2014in this case, question words and column\/table names in the schema. While effective in multiple fields (opens in new tab)<\/span><\/a>, self-attention requires large training corpora, and human-authored DBQA datasets reach only up to 10,000 training instances. With limited data, even simple natural language relations, easily discovered in machine translation and other NLU systems, can be challenging. For instance, in our DBQA experiments, standard Transformers struggled to reliably link \u201csales\u201d<\/em> to the column sale_amount<\/code>. In the 2020 Meeting of the Association for Computational Linguistics (ACL)<\/a> publication \u201cRAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers,\u201d<\/a> we introduced relation-aware Transformers (RAT)<\/em>, which reduce training data requirements by incorporating known relational information about the database into self-attention. They allow the encoder to consider, for instance, database foreign keys without either rediscovering them (as in standard Transformers) or hard-coding the network\u2019s relational structure to follow them (as in graph neural networks). As such, RAT augments the learning efficacy of Transformers with rich background knowledge about relational structure.<\/p>\n\n\n\n

We combine a RAT encoder with a grammar-driven program decoder into an end-to-end DBQA model called RAT-SQL. On Spider (opens in new tab)<\/span><\/a>, currently the most challenging text-to-SQL dataset, RAT-SQL achieved a new state of the art of 65.6 percent exact-match accuracy at the time of its publication. Relation-aware data contextualization added a more than 5 percent margin over previous best techniques. Since then, numerous researchers from other institutions have built upon RAT-SQL in their DBQA models, achieving even more impressive performance (opens in new tab)<\/span><\/a>.<\/p>\n\n\n\n

StruG: Structure-grounded pretraining for robust language understanding<\/h3>\n\n\n\n

Joint representation of the question and its data context fundamentally requires solving an alignment problem<\/em>\u2014that is, linking words in the question to the data columns they reference\u200b. Relation-aware Transformers effectively incorporate known relations between the question and the data, but alignment often requires additional background knowledge either from the database content or from NLU at large. For example, in Figure 2, linking \u201cHistory\u201d<\/em> to department_name<\/code> is challenging without looking at the database content even though people\u2019s real-world experience intuitively aligns these two phrases.<\/p>\n\n\n\n

\"Two
Figure 2: For a task-oriented conversational system to account for data context, it must be able to link words in a query to their corresponding data columns based on background knowledge about the relations between the question and the data. The relations align words in a query to column names or table content, as illustrated here in database question answering (top) and in parallel text-table corpora from the web (bottom).<\/figcaption><\/figure>\n\n\n\n

Background knowledge on question-table alignment naturally occurs in parallel text-table corpora such as ToTTo (opens in new tab)<\/span><\/a>. They pair data tables with relevant utterances about them, such as table summaries and data references. While such utterances are not typically in question form, they exhibit the same alignment patterns as questions in DBQA. In an upcoming 2021 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) (opens in new tab)<\/span><\/a> publication \u201cStructure-Grounded Pretraining for Text-to-SQL,\u201d<\/a> we propose a method to leverage this data for pretraining contextual representation models. StruG, short for \u201cStru<\/strong>cture-G<\/strong>rounded pretraining,\u201d introduces three critical pretraining tasks that use text-table alignment annotations as weak supervision:<\/p>\n\n\n\n