Research Group Projects and Descriptions

The Software Agents group investigates a new paradigm for software that acts like an assistant to a user of an interactive interface rather than simply as a tool. While not necessarily as intelligent as a human agent, agent software can learn from interaction with the user, and proactively anticipate the user's needs. We build prototype agent systems in a wide variety of domains, including text and graphic editing, Web browsing, e-commerce, information visualization, and more.

Current experiments in agent software rely mostly on domain-specific applications that either have been programmed from scratch, or explicitly modified with an agent in mind. Is it possible to make a tool kit or protocol that would allow an agent to communicate and to control applications that have been constructed more conventionally? Can the agent "take the place" of the user in the interface? Can the agent have access to the application's data and behavior? Will commercial "inter-application communication" mechanisms suffice? What is the division of labor between the agent and the application? This work will explore these questions.

Effective image annotation and retrieval is bound up with image use. In this project, annotation, retrieval, and use are integrated, facilitating the finding and using images. A proactive user-interface agent seeks chances for image annotation and retrieval in the context of the user's everyday work, using an agent that sit in the user's text editor or other application and continuously monitors typing. Searches are automatically performed from an image library, and images relevant to the current text can be inserted in a single operation. Descriptions of images for storytelling can be seamlessly employed as raw material for annotation. Common-sense knowledge about situations in which pictures are taken, described, or used can help provide semi-automatic annotation and indirect inference for retrieval. Our approach does not completely automate the annotation/retrieval process, but it does reduce user-interface overhead, leading to better-annotated image libraries and fewer missed opportunities for image use.

Alumni Contributor(s): Xinyu H. Liu and Kim Waters

AnalogySpace enables common-sense reasoning through principal component analysis. It projects the information in ConceptNet into a reduced-dimensional space that describes common-sense concepts and their properties in terms of automatically discovered correlations called "eigenconcepts." AnalogySpace can be used to infer new information, reason about ad hoc categories, detect topics in text, and compare concepts on scales that can be generated on the fly.

The Open Mind Common Sense project collects its knowledge base from ordinary people. Acquiring useful knowledge from untrained volunteers requires asking them the right questions and keeping them interested, and we use a variety of online interfaces and games to do so. We present some of these interfaces that enable people to teach computers what they know.

Common sense enables us to build innovative recommendation systems that are more interactive and user-friendly than traditional collaborative filtering systems. By applying ideas from blending and PerspectiveSpace to recommendations, we discover the characteristics of products that drive user ratings. We can use these characteristics to build intelligent recommendation agents and effective product exploration tools.

This project aims to develop an intelligent personal-finance advisory agent that bridges the gap between the novice user and the expert model of the finance domain. The agent uses common-sense reasoning and inference for associating the user's personal life, financial situation, and goals with the attributes of the expert domain model and vice versa. The agent interface provides a natural-language interface for elicitation and explanations of design and process rationale. The architecture of the system is domain-independent and consequently can be used for any novice-expert domain model.

Alumni Contributor(s): Ashwani Kumar

A long-standing dream of artificial intelligence has been to put common-sense knowledge into computers–enabling machines to reason about everyday life. Some projects, such as Cyc, have begun to amass large collections of such knowledge. However, it is widely assumed that the use of common sense in interactive applications will remain impractical for years, until these collections can be considered sufficiently complete, and common-sense reasoning sufficiently robust. Recently we have had some success in applying common-sense knowledge in a number of intelligent interface agents, despite the admittedly spotty coverage and unreliable inference of today's common-sense knowledge systems.

Alumni Contributor(s): Xinyu H. Liu and Push Singh

We have developed, Common Consensus: a fun, self-sustaining web-based game, that both collects and validates Commonsense knowledge about everyday goals. Goals are a key element of commonsense knowledge; in many of our inferface agents, we need to recognize goals from user actions (plan recognition), and generate sequences of actions that implement goals (planning). We also often need to answer more general questions about the situations in which goals occur, such as when and where a particular goal might be likely, or how long it is likely to take to achieve.

Alumni Contributor(s): Push Singh

We are developing next-generation architectures for artificial intelligence based on Professor Minsky's "Society of Mind" theory of human thinking. The main idea is that the key to human flexibility and resourcefulness is mental diversity: we have many ways to solve every kind of problem; when we get stuck trying one method of solution, we can switch to another. We are exploring how this idea can be applied at different places and levels in a cognitive architecture, in order to build systems capable of robust common-sense reasoning.

Alumni Contributor(s): Push Singh

Imparting common-sense knowledge to computers enables a new class of intelligent applications better equipped to make sense of the everyday world and assist people with everyday tasks. While previous attempts have been made to acquire and structure common-sense knowledge, they have either been inadequate in capturing the breadth of knowledge needed for the enterprise, or their complicated representation schemes have made them difficult to incorporate into applications. Our approach to this problem is ConceptNet, a freely available common-sense knowledge base that possesses a great breadth of knowledge that can be easily incorporated into applications. Built from the Open Mind Common Sense corpus, which acquires common-sense knowledge from a Web-based community of instructors, ConceptNet is a semantic network of 1.6 million items of common-sense knowledge, and a set of tools for making inferences using this knowledge.

Alumni Contributor(s): Xinyu H. Liu and Push Singh

We have developed technology that enables easy analysis of semantic data, blended in various ways with common-sense world knowledge. The results support reasoning by analogy and association. A packaged library of code is being made available to all sponsors.

One of the biggest challenges for the digital economy is what to do when things go wrong. Orders get misplaced, numbers mistyped, requests misunderstood: then what? Consumers are frustrated by long waits on hold, misplaced receipts, and delays to problem resolution; companies are frustrated by the cost of high-quality customer service. Online companies want customers’ trust, and how a company handles problems directly impacts that. We explore how software agents and other technologies can help with this issue. Borrowing ideas from software debugging, we can have agents help to automate record-keeping and retrieval, track dependencies, and provide visualization of processes. Diagnostic problem-solving can generate hypotheses about causes of errors, and seek information that allows hypotheses to be tested. Agents act on behalf of both the consumer and the vendor to resolve problems more quickly and at lower cost.

Alumni Contributor(s): Earl Wagner, Ethan Mollick and Tom Stocky

Story understanding is a notoriously difficult problem in AI. Broad-spectrum, common-sense knowledge about the world is a good resource, but current common-sense knowledge bases are far from human-level story understanding. We examine affective story understanding in order to perceive the broad emotional overtones of a story at the sentence level, using both a common-sense perspective and the observation that much of the way we emote in response to everyday situations is part of a shared human experience and therefore a part of common sense. With a corpus of common-sense knowledge, we create a semantic network of everyday situations and the emotions associated with them, which, when combined with our linguistic processing, lets our system classify story sentences into six primitive emotions. We then explore how this technology enables innovations in emotional UIs such as EmpathyBuddy, or in prosody, emotional TTS, gaming, story evaluation, and emotional indexing of documents.

Alumni Contributor(s): Xinyu H. Liu and Ted Selker

Because common-sense knowledge differs culturally, misunderstandings frequently occur. Because differences can be subtle, there has been little work in trying to detect places in text where cultural differences might arise. We explicitly represent the common-sense knowledge of each culture in separate knowledge bases. By analyzing a text, we can find differences between each culture's knowledge concerning its subject. For example, given an invitation to a party, the system is able to infer that in an American cultural context, hip-hop dancing might be expected, but in a Mexican context, salsa dancing might be the norm. We are building an email client that suggests knowledge from multiple cultures that might be relevant, while watching the user's typing.

Alumni Contributor(s): Jose H. Espinosa

Consumer electronics devices are becoming more complicated, intimidating users. These devices do not know anything about everyday life or human goals, and they show irrelevant menus and options. Using common-sense reasoning, we are building a system, Roadie, with knowledge about the user's intentions; this knowledge will help the device to display relevant information to reach the user's goal. For example, an amplifier should suggest a play option when a new instrument is connected, or a DVD player suggest a sound configuration based on the movie it is playing. This will lead to more human-like interactions with these devices. We have constructed a Roadie interface to real consumer electronics devices: a television, set top box, and smart phone. The devices communicate over Wi-Fi, and use the UPnP protocols.

Alumni Contributor(s): Jose H. Espinosa

How can computers learn from text when reading itself requires knowledge? Human writings are semantically compressed and rely on common-sense knowledge to complete omitted details. To use common-sense knowledge bases for machine reading, current techniques fall short of retrieving only knowledge relevant to a particular text. We are associating the semantic knowledge with procedural plan representations, making the language-understanding problem a two-step problem of plan recognition and automated planning, where plans guide which inferences should be made about the text. We begin by learning a rich plan representation from a parallel corpus of common-sense stories. We use a collection of English narratives describing the steps required to accomplish everyday domestic tasks. Because the corpus contains many different versions of how to accomplish a given task, we must recognize incomplete descriptions, infer semantic equivalence of different predicate-argument structures, and detect and represent different ways to accomplish the same goal.

This project explores how modern graphical interface techniques and explicit support for the user's problem-solving activities can make more productive interfaces for debugging, which accounts for half the cost of software development. Animated representations of code, a reversible control structure, and instant connections between code and graphical output are some of the techniques used.

Technical documentation for hardware and software is expensive to produce, and often inaccurate and inadequate. We are exploring a new approach to producing technical documentation in which an expert interacts with a simulation of a device, and the system automatically produces both written English descriptions and visual illustrations.

It's well known that living in a foreign country dramatically improves the effectiveness of learning a second language over classroom study alone. This is likely because people make associations with the foreign language as they see and participate in everyday life activities. We are designing language-teaching sequences for a sensor-equipped residence that can detect user interaction with household objects. We use our common-sense knowledge base and reasoning tools to construct teaching sequences, wholly in the target language, of sentences and question-answering interactions that gradually improve the learner's language competence. For example, the first time the user sits in a chair, the system responds with the foreign-language word for "chair," and later with statements and questions such as, "You sit in the chair" (complete sentence), "You sat in the chair" (tenses), "What is the chair made of?" (question, materials), or "Why are you sitting in the chair?" (goals, plans).

Recognition-based technology has made substantial advances in the past few years because of enhanced algorithms and faster processing speeds. However, current recognition systems are still not reliable enough to be integrated into user interface designs. A possible solution to this problem is to combine results from existing recognition systems and mutually disambiguate the unreliable sections. Piecing together partial results obtained from each mode of recognition can derive more reliable results. In addition, the results of one recognition system can be used to prepare the other recognition system. We are experimenting with an approach that uses a software agent to integrate off-the-shelf recognition applications via scripting languages. We use a software agent called MARCO (Multimodal Agent for Route Construction) that utilizes multiple recognition systems to assist users in giving directions for urban navigation.

A ConceptNet in English is already established and working well. We are now attempting to expand it to other languages and cultures. This project is an extended ConceptNet with Korean common sense, which is fundamentally different from English. Through this project, we can learn how to expand the ConceptNet into other languages and how to connect them. By connecting English and Korean ConceptNets, we are hoping not only to see cultural or linguistic differences, but also to solve problems such as the ambiguity of multivocal words, which were difficult to solve with only one ConceptNet.

This project aims to collect and reason over common-sense knowledge in languages other than English. We have collected large bodies of common-sense knowledge in Portuguese and Korean, and we are expanding to other languages such as Spanish, Dutch, and Italian. We can use techniques based on AnalogySpace to discover correlations between languages, enabling our knowledge bases in different languages to learn from each other.

Alumni Contributor(s): Hyemin Chung, Jose H. Espinosa, Wonsik Kim and Yu-Te Shen

How would you browse a VERY large display space, such as a street map of the entire world? The traditional solution is zoom and pan, but these operations have drawbacks that have gone unchallenged for decades. Shifting attention loses the wider context, leading to that "lost in hyperspace" feeling. We are exploring alternative solutions, such as a new technique that allows zooming and panning in multiple translucent layers.

We present a way of infusing any dataset with common sense. When domain-specific data is combined with the general knowledge in ConceptNet, new ways of organizing, visualizing, and reasoning over the data emerge. In domains such as consumer lifestyle modeling, knowledge acquisition from free text, and personal financial management, most information based on natural language can benefit from a little added common sense.

The Open Mind Common Sense project has collected hundreds of thousands of statements of common-sense knowledge from volunteers on the Internet, using a variety of online activities in several different languages. Open Mind Commons aims to use analogical reasoning to make connections between similar ideas while highlighting the relevant differences as well. These analogies can give a computer a better understanding of the relationships between objects, situations, and cultures. It is often difficult to search through and coordinate lexical information across data sources, each of which has its own separate interface and viewing software. We have approached this problem by creating a unified, flexible interface for various natural-language processing resources.

Alumni Contributor(s): Hyemin Chung

Words mean different things to different people, and capturing these differences is often a subtle art. This project is for the development of a system for discovering distinct communities of people with distinct jargon usage or belief structures from simple rating data on common-sense knowledge. PerspectiveSpace is an approach whereby elementary linear operations are used to perform calculations on user models and microtheories.

Skilled programmers can learn a great deal about almost any subject by developing a program to analyze or simulate a phenomenon. However, too often the tedium of representing their ideas in popular programming languages takes attention away from what they are studying, to the point that many scientists (or aspiring scientists) don't even bother. We are investigating ways of focusing the programming interaction on the phenomenon, through exploring new representations of programs and ways to interactively develop these relationships based on observed examples.

We want to build programming systems that can converse with their users to build computer programs. Such systems will enable users without programming expertise to write programs using natural language. The text-based, virtual-world environments called the MOO (multi-user, object-oriented Dungeons and Dragons) allow their users to build objects and give them simple, interactive, text-based behaviors. These behaviors allow other participants in the environment to interact with those objects by invoking actions and receiving text messages. Through our natural-language dialog system, the beginning programmer will be able to describe objects and the messages in MOO environments.

Analogy is a powerful comparison mechanism, commonly thought to be central to human problem solving. Analogies like "an atom is like the solar system" enable people to effectively transfer knowledge to new domains. Can we enable computers to do similar comparisons? Prior work on analogy (structure mapping) provides guidance about the nature of analogies, but implementations of these theories are inefficient and brittle. We are working on a new analogy mechanism that uses instance learning to make robust, efficient comparisons.

Today, people can tell stories by composing, manipulating, and sequencing individual media artifacts using digital technologies. However, these tools offer little help in developing a story's plot. Specifically, when a user tries to construct her story based on a collection of individual media elements (videos, audio samples), current technological tools do not provide helpful information about the possible narratives that these pieces can form. Storied Navigation is a novel approach to this problem; media sequences are tagged with free-text annotations and stored as a collection. To tell a story, the user inputs a free-text sentence and the system suggests possible segments for a storied succession. This process iterates progressively, helping the user to explore the domain of possible stories. The system achieves the association between the input and the segments' annotations using reasoning techniques that exploit the WordNet semantic network and common-sense reasoning technology.

Alumni Contributor(s): Barbara Barry

ToDoGo is a system for managing to-do lists on mobile devices. ToDoGo understands how to-do list entries relate to a user's everyday life activities, and can give them location-aware help in scheduling events and finding places to accomplish tasks.