Introduction

In fact, “semantic parsing” is, ironically, a semantically ambiguous term, which could refer to:

• Semantic role labeling (also known as shallow semantic parsing, see previous post for more information).
• Finding generic relations in text.
• Transforming a natural language sentence into its meaning representation.

In this post, semantic parsing is the task of mapping natural language sentences (NL) into complete formal meaning representation (MRs) which a computer can execute for some domain -specific application ¹. Representing the meaning of natural language is ultimately a difficult philosophical question and many attempts have been made to define generic formal semantics of natural language, including:

• Abstract Meaning Representation (AMR)

• Semantic Dependency Parsing (SDP)

• Meaning Representation Language (MRL)

• SQL Query

Abstract Meaning Representation

Abstract Meaning Representation (AMR) ² is a semantic formalism in which the meaning of a sentence is encoded as a rooted, directed, acyclic graph. Nodes represent concepts, and labeled directed graph edges represent the relationships between them. For example, for the sentence “the boy wants to believe the girl” we can have the corresponding AMR graph (in penman format) as:

(w / want-01
	:ARG0 (b / boy)
	:ARG1 (b2 / believe-01
		:ARG0 (g / girl)
		:ARG1 b))

AMR parsing, the task of transforming a sentence into its AMR graph, is a challenging task as it requires the parser to learn to predict not only concepts, which consist of predicates, lemmas, named entities, wiki-links and co-references, but also a large number of relation types based on the semantic role in PropBank (see the previous blog Semantic Role Labeling if you are interested).

Data

There are multiple version of AMR data, including:

• LDC2013E117: 10,853 sentences with 13,050 AMRs (corpus includes multiple annotations for some sentences)

• LDC2014T12 (AMR 1.0): 13,051 sentences.

• LDC2015E86: 19,572 sentences. This version includes more AMRs, wikification, AMR adoption of new unified PropBank frames, AMR deepening, automatic AMR-English alignments, and correction of AMR annotation errors.

• LDC2016E25: 39,260 sentences.

• LDC2017T10 (AMR 2.0): 39,260 sentences.

Experiment & Results

These are the AMR parsing results on LDC2014T12, LDC2015E86 and LDC2017T10 test sets, where 2014N refers to the newswire section of LDC2014T12. You should look the original paper if you are interested in the model. Please contact me (through email) if the statistics is incorrect.

Semantic Dependency Parsing

Task 8 at SemEval 2014 defines Broad-Coverage Semantic Dependency Parsing (SDP) as the problem of recovering sentence-internal predicate-argument relationships for all content words, i.e., the semantic structure constituting the relational core of sentence meaning ³.

SDP seeks to stimulate the dependency parsing community to move towards more general graph processing, to thus enable a more direct analysis of Who did What to Whom? In addition to its relation to syntactic dependency parsing, the task also has some overlap with Semantic Role Labeling (SRL) ⁴.

The SDP parsers are required to identify all semantic dependencies, i.e., compute a representation that integrates all content words in one structure. Different from SRL, SDP task does not encompass predicate disambiguation, a design decision in part owed to SDP’s goal to focus on parsing-oriented, i.e., structural, analysis, and in part to lacking consensus on sense inventories for all content words.

SDP uses three distinct target representations for semantic dependencies. Including:

DM: DELPH-IN MRS-Derived Bi-Lexical Dependencies

These semantic dependency graphs originate in a manual re-annotation of Sections 00–21 of the WSJ Corpus with syntactico-semantic analyses derived from the LinGO English Resource Grammar (ERG) ⁵. The DM target representations are derived through a two-step ‘lossy’ conversion of MRSs, first to variable-free Elementary Dependency Structures (EDS) ⁶, then to ‘pure’ bi-lexical form - projecting some construction semantics onto word-to-word dependencies ⁷. For example, for the sentence “Ms. Haag plays Elianti.”, we have the DM representation as follows:

PAS: Enju Predicate-Argument Structures

The Enju parsing system is an HPSG-based parser for English. The grammar and the disambiguation model of this parser are derived from the Enju HPSG treebank, which is automatically converted from the phrase structure and predicate-argument structure annotation of the PTB. The PAS data set is extracted from the WSJ portion of the Enju HPSG treebank. While the Enju treebank is annotated with full HPSG-style structures, only its predicate-argument structures are convert into the SDP data format for use in the SDP task.

PCEDT: Prague Tectogrammatical Bi-Lexical Dependencies

The Prague Czech-English Dependency TreeBank (PCEDT) ⁸ is a set of parallel dependency trees over the WSJ texts from the PTB, and their Czech translations. The specifics of the PCEDT representations are best observed in the procedure that converts the original PCEDT data to the SDP data format; see Miyao et al. 2014 ⁹.

Data

• Semeval-2014 Task 8: For this task, there will be three data sets: training, development, and test. On November 4, 2013, a sample (of some 190 dependency graphs) from the training data was published as trial data, demonstrating key characteristics of the task. Since December 13, some 750,000 tokens of annotated text are available as training data; please subscribe to the task mailing list for access information.

• Semeval-2015 Task 8: This Task is a re-run with some extensions of Task 8 at SemEval-2014.

Experiment & Results

The results are the labeled parsing performance (F1 score) on both in-domain and out-of-domain test data and are based on the English dataset from SemEval 2015 Task 18 closed track. The data split is 33,964 training sentences from 00-19 of the WSJ corpus, 1,692 development sentences from 20, 1,410 sentences from 21 as in domain test data, and 1,849 sentences sampled from the Brown Corpus as out-of-domain test data.

In-Domain

Out-Of-Domain

Logical Form Representation

First-order logic is used as a semantic representation language (MRL), which is designed by the creator of the application to suit the application’s need independent of natural language. Some MRL data sets are based on Prolog database, where two well-known semantic parsing benchmark data sets are Jobs and Geo ¹⁰.

Here we explain the features of the Geoquery representation language through a sample query:

Input: "What is the largest city in Texas?"
Query: answer(C, largest(C, (city(C), loc(C, S), const(S, stateid(texas)))))

Objects are represented as logical terms and are typed with a semantic category using logical functions applied to possibly ambiguous English constants (e.g. stateid(Mississippi), riverid(Mississippi)). Relationships between objects are expressed using predicates; for instance, loc(X, Y) states that X is located in Y.

We also need to handle quantifiers such as ‘largest’. We represent these using meta-predicates for which at least one argument is a conjunction of literals. For example, largest(X, Goal) states that the object X satisfies Goal and is the largest object that does so, using the appropriate measure of size for objects of its type (e.g. area for states, population for cities). Finally, an unspecified object required as an argument to a predicate can appear elsewhere in the sentence, requiring the use of the predicate const(X, C) to bind the variable X to the constant C.

Data

• Geo 880: A set of 880 queries to a database of U.S. geography containing basic information about the U.S. states like population, area, capital city, neighboring states, and so on. It is originally annotated with Prolog style semantics. Geo has 880 instances split into a training set of 680 training examples and 200 test examples ¹¹.

• Jobs 640: A set of 640 queries to a database of computer-related job postings, such as job announcements, from the USENET newsgroup austin.jobs. Information from these job postings are extracted to create a database which contains the following types of information: 1) the job title, 2) the company, 3) the recruiter, 4) the location, 5) the salary, 6) the languages and platforms used, and 7) required or desired years of experience and degrees. The training-test split contains 500 training and 140 test instances.

• ATIS: A set of 5,410 queries to a flight booking system, and the representations are of lambda-calculus logical forms. The ATIS database consists of data obtained from the Official Airline Guide (OAG, 1990), organized under a relational schema. The database remained fixed throughout the pilot phase. It contains information about flights, fares, airlines, cities, airports, and ground services, and includes twenty-five supporting tables. The large majority of the questions posed by subjects can be answered from the database with a single relational query.

Experiment & Results

The results are based on accuracies which are defined as the proportion of the input sentences that are correctly parsed to their gold standard logical forms.

SQL Query

In recent years, there has been considerable interest in automated synthesis of programs from informal specifications. A particularly promising application domain for such computer-aided programming techniques is the automated synthesis of database queries. Although many end-users need to query data stored in some relational database, they typically lack the expertise to write complex queries in declarative query languages such as SQL. As a result, there has been a flurry of interest in automatically synthesizing SQL queries from informal specifications. Here we will talk about the techniques that can generate SQL queries from natural language (NL) descriptions.

For example, for the sentence “how many CFL teams are from York College”, we will have the SRL query like:

SELECT COUNT CFL Team FROM
CFLDraft WHERE College = "York"

Data

• WikiSQL: WikiSQL is a collection of questions, corresponding SQL queries, and SQL tables. It is the largest hand-annotated semantic parsing dataset to data - it is an order of magnitude larger than other datasets that have logical forms, either in terms of the number of examples or the number of tables. The corpus consists of 80654 hand-annotated instances of natural language questions, SQL queries, and SQL tables extracted from 24241 HTML tables from Wikipedia ¹².

Experiment and Results

To learn more about the experiment and results on WikiSQL, see https://github.com/salesforce/WikiSQL for more details.