Representation:

The Empty Elements Project

Empty elements are elements left out by the speaker and inferred by the hearer. They are central to obtaining human-like language understanding. Some examples of empty elements are:

• Fused heads: I bought three apples and ate two [apples].
• Verb-phrase ellipsis: John bought books, Jane didn’t [buy books].
• Aspectual verb constructions: I finished [reading? writing?] the book.
• Bridging: I love these shoes. The color [of the shoes] is beautiful!.

In this project we investigate Empty Elements Expansion (EEE), a challenge wherein we aim to automatically recognize and resolve such empty elements in texts.

Bridging, for example, is a (non-identity) implicit relation between two nouns in the text. For example in the sentence “I entered the room. The ceiling was high” there is an implicit part-whole relation between “the ceiling” and “the room”, yielding the “complete” expression “the ceiling [of the room]”. How can we collect data concerning those missing elements? How can we design learning models to infer them? How can these empty elements aid Natural Language Understanding applications, and in particular, Natural Language Programming?

For more detail about bridging and the data collection interface, check out our introduction and qualification tests and our annotation tool.

Models:

Natural Language Navigation

Following navigation instructions in natural language requires a composition of language, action, and knowledge of the environment. Knowledge of the environment may be provided via visual sensors or as a symbolic world representation referred to as a map. This project aims at learning to ground objects using multimodal world knowledge. This project has been selected this year (2020) to receive funds from Google Faculty Research Award.

For materials, data sets and models, follow us.

Applications:

NLPRO-CT: Nurturing Computational Thinking Skills with Natural Language Programming

In the NLPRO-CT we develop and use NLPRO semantic parsers to build accessible educational apps that enhance young students’ Computational Thinking (CT) skills.

The Hexagon Board App is our first app being developed in this framework. In this conversational app users verbally instruct the automatic painter how to draw their desired image on a kids board game paved with hexagonal tiles. To reach that goal we created two complementary apps to collect the initial dataset required to train our model. The dataset is comprised of corresponding pairs of images and drawing procedure. The first version is used to create those pairs whereas the second one is used to ensure the quality of those pairs.

Read about and experiment with both apps:
Introduction & Hexagon Board App 1
Introduction & Hexagon Board App 2

Take part in our project and contribute new and interesting images to our dataset.