Research Group Projects and Descriptions

The Affective Computing research group aims to bridge the gap between computational systems and human emotions. Our research addresses machine recognition and modeling of human emotional expression, machine learning of human preferences as communicated by user affect, intelligent computer handling of human emotions, computer communication of affective information between people, affective expression in machines and computational toys, emotion modeling for intelligent machine behavior; tools to help develop human social-emotional skills, and new sensors and devices to help gather, communicate, and express emotional information.

Affect as Index is a tool that takes group physiological data as input, aggregates the data across different demographic dimensions, and attaches them to media content. Users can review videotaped or prerecorded events by clicking on points of interest in a physiological graph. This software addresses two challenges: 1) the difficulty of expressing and sharing emotions with others, and 2) the laborious task of monitoring interpersonal interactions within natural settings. For the former, groups interested in discussing shared and dissimilar emotions evoked during experiences can use this tool to place context around their dialogue. For the latter, "meaningful moments" that are observed within natural interactions can be marked and these moments can be superimposed on the physiological data collected. In this way, affect and observations of affect can be used to index group-level significant moments that occur within volumes of video data.

Recent findings in affective neuroscience and psychology indicate that human affect and emotional experience play a significant and useful role in human learning and decision-making. Most machine-learning and decision-making models, however, are based on old, purely cognitive models, and are slow, brittle, and awkward to adapt. We aim to redress many of these classic problems by developing new models that integrate affect with cognition. Ultimately, such improvements will allow machines to make smarter and more human-like decisions for better human-machine interaction.

Alumni Contributor(s): Alea Teeters

Companies would like more new products to be successful in the marketplace, but current evaluation methods such as focus groups do not accurately predict customer decisions. We are developing new technology-assisted methods to try to improve the customer-evaluation process and better predict customer decisions. The new methods involve multi-modal affective measures (such as facial expression and skin conductance) together with behavioral measures, anticipatory-motivational measures, and self-report cognitive measures. These measures are combined into a novel computational model, the form of which is motivated by findings in affective neuroscience and human behavior. The model is being trained and tested with customer product evaluations and marketplace outcomes from real product launches.

People who have difficulty communicating verbally (such as many people with autism) sometimes send nonverbal messages that do not match what is happening inside them. For example, a child might look calm and receptive to learning, while having a heart rate of over 120 bpm and being on the verge of a meltdown or shutdown. This mismatch can lead to serious problems, including misunderstandings such as "he became aggressive for no reason." We are creating new technologies to address this fundamental communication problem and enable the first long-term, ultra-dense longitudinal data analysis of emotion-related physiological signals. We hope to equip individuals with personalized tools to understand the influences of their physiological state on their own behavior (e.g., "which state helps me best maintain my attention and focus for learning?"). The data from daily life will also advance basic scientific understanding of the role of autonomic nervous system regulation in autism.

Alumni Contributor(s): Hoda Eydgahi

Social-emotional communication difficulties lie at the core of autism spectrum disorders, making interpersonal interactions overwhelming, frustrating, and stressful. We are developing the world's first wearable affective technologies to help the growing number of individuals diagnosed with autism—approximately 1 in 150 children in the United States—learn about nonverbal communication in a natural, social context. We are also developing technologies that build on the nonverbal communication that individuals are already using to express themselves, to help families, educators, and other persons who deal with autism spectrum disorders to better understand these alternative means of nonverbal communication.

Alumni Contributor(s): Alea Teeters

To help people improve their reading of faces during natural conversations, we developed a video tool to evaluate this skill. First, we collected over 100 videos of conversations between pairs of both autistic and neurotypical people, each of whom wore a Self-Cam. Next, the videos were manually segmented into chunks of 7-20 seconds according to expressive content, labeled, and sorted by difficulty—all tasks we plan to automate using technologies under development. Next, we built a rating interface including videos of self, peers, familiar adults, strangers, and unknown actors, allowing for performance comparisons across conditions of familiarity and expression. We obtained reliable identification (by coders) of categories of smiling, happy, interested, thinking, and unsure in the segmented videos. The tool was finally used to assess recognition of these five categories for eight neurotypical and five autistic people. Results show some autistics approaching the abilities of the neurotypicals while several score just above random.

Alumni Contributor(s): Alea Teeters

We propose a set of customizable, easy-to-understand, and low-cost physiological toolkits in order to enable people to visualize and utilize autonomic arousal information. In particular, we aim for the toolkits to be usable in one of the most challenging usability conditions: helping individuals diagnosed with autism. This toolkit includes: wearable, wireless, heart-rate and skin-conductance sensors; pendant-like and hand-held physiological indicators hidden or embedded into certain toys or tools; and a customized software interface that allows caregivers and parents to establish a general understanding of an individual's arousal profile from daily life and to set up physiological alarms for events of interest. We are evaluating the ability of this externalization toolkit to help individuals on the autism spectrum to better communicate their internal states to trusted teachers and family members.

People express and communicate their mental states, including emotions, thoughts, and desires through facial expressions, vocal nuances, gestures, and other nonverbal channels. We have developed a computational model that enables the real-time analysis, tagging, and inference of cognitive-affective mental states from facial video. This framework combines bottom-up, vision-based processing of the face (e.g., a head nod or smile) with top-down predictions of mental-state models (e.g., interest and confusion) to interpret the meaning underlying head and facial signals over time. Our system tags facial expressions, head gestures, and affective-cognitive states at multiple spatial and temporal granularities in real time and offline, in both natural human-human and human-computer interaction contexts. The system is being made available on multiple platforms, including portable devices. Applications range from measuring people's experiences to a training tool for autism spectrum disorders.

Emotions are often conveyed through gesture. Instruments that respond to gestures offer musicians new, exciting modes of musical expression. This project gives musicians wireless, gestural-based control over guitar effects parameters. For example, with this system, a guitarist can manipulate any style of pitch bending (from subtle vibrato, to whole step bends, to two-octave dive bombs) just by moving the headstock of the guitar.

In this research, a proactive emotional health system, geared toward supporting emotional self-awareness and empathy, was built as a part of a long-term research plan for understanding the role digital technology can play in helping people to reflect on their beliefs, attitudes, and values. The system, G.I.R.L.S. (Girls Involved in Real-Life Sharing), allows users to reflect actively upon the emotions related to their situations through the construction of pictorial narratives. The system employs common-sense reasoning to infer affective content from the users' stories and support emotional reflection. Users of this new system were able to gain new knowledge and understanding about themselves and others through the exploration of authentic and personal experiences. Currently, the project is being turned into an online system for use by school counselors.

Alumni Contributor(s): Ted Selker

We are developing a tiny wearable wireless sensor platform that allows comfortable, long-term sensing of physiological information coupled with low-cost connectivity to consumer devices including mobile phones and the XO laptop. This platform has many applications, including health monitoring for outpatients or elderly, communication of affective information for people who are non-speaking or otherwise interested in sharing this information, education for individuals who want to learn about their own internal physiological changes during daily life, and customer experience data gathering in mobile situations.

Alumni Contributor(s): Hoda Eydgahi and Clayton Williams

This project develops new theory and algorithms to enable computers to make rapid and accurate inferences from multiple modes of data, such as determining a person's affective state from multiple sensors—video, mouse behavior, chair pressure patterns, typed selections, physiology, and more. Recent efforts focus on understanding the level of a person's attention, which is useful for things such as determining when to interrupt. Our approach is Bayesian: formulating probabilistic models on the basis of domain knowledge and training data, and then performing inference according to the rules of probability theory. This type of sensor fusion work is especially challenging because of the problems of sensor channel drop-out, different kinds of noise in different channels, dependence between channels, scarce and sometimes inaccurate labels, and patterns to detect that are inherently time-varying. We have constructed a variety of new algorithms for solving these problems and demonstrated their performance gains over other state-of-the-art methods.

Alumni Contributor(s): Ashish Kapoor

People on the autism spectrum face a number of challenges, including motor movement issues that can cause limbs to cease activity. Circumstantial evidence suggests that autonomic nervous system influences related to stress and overload may arise from and contribute to these problems. We propose to allow individuals to monitor several physiological parameters to see if there are patterns that recognize or predict the onset of their individual motor problems. We plan to develop new, wearable technology to treat these problems via the use of tiny, vibrotactile devices carefully placed at the joints. We hypothesize that some methods of touch-feedback and vibration at the joints may enable individuals to recover motor functioning during episodes of intermittent loss. We are also exploring the development of personally controlled devices that facilitate finer motor movement for augmenting communication as needed for assisting in typing or pointing.

We have developed a device that can wirelessly detect a beating heart over a short distance (1 meter). In addition to medical/health applications, his device was designed for security and safety applications for automobile/truck drivers as well as ATM machines.

We have developed a novel market game, "Prediction Game and Experience Sharing" (PreGES, pronounced as PreGuess), that harnesses people's collective prediction and experience sharing to forecast the success or failure of new items (e.g., products, services, UI designs). Companies can register their new items on this market (as a testbed) to ask people's collective opinion. In each PreGES trial session, a participant makes his or her own best prediction on other people's overall opinions about the new items to get incentives (e.g., real opportunities to experience the items) and have fun in gambling-like games. As a participant’s guess (or portfolio) approaches the collective guess of all participants, he or she has a greater chance of winning an incentive. Participants improve the accuracy of their next prediction by sharing experiences. As participants have more trial sessions, their collective prediction converges into one common opinion (forecasting the success or failure of new items).

A robotic computer that moves its monitor "head" and "neck," but that has no explicit face, is being designed to interact with users in a natural way for applications such as learning, rapport-building, interactive teaching, and posture improvement. In all these applications, the robot will need to move in subtle ways that express its state and promote appropriate movements in the user, but that don't distract or annoy. Toward this goal, we are giving the system the ability to recognize states of the user and also to have subtle expressions.

Alumni Contributor(s): Ashish Kapoor and Karen Liu

A small number of studies support the notion that there is a functional relationship between movement stereotypy and arousal in individuals with ASD, such that changes in autonomic activity either precede or are a consequence of engaging in stereotypical motor movements. Unfortunately, however, it is difficult to generalize these findings since previous studies fail to report reliability statistics that demonstrate accurate identification of movement stereotypy start and end times, and use autonomic monitors that are obtrusive and thus only suitable for short-term measurement in laboratory settings. The current investigation further explores the relationship between movement stereotypy and autonomic activity in persons with autism by combining state-of-the-art ambulatory heart rate monitors to objectively assess arousal across settings and wireless, wearable motion sensors and pattern recognition software that can automatically and reliably detect stereotypical motor movements in individuals with autism in real time.

SmileSeeker is a novel, machine-vision system that captures and provides quantified information about nonverbal communication where social interactions naturally happen. For example, in banking services, bank tellers can already observe the facial expressions, head gestures, and eye gaze of their customers, but this tool lets them observe their own expressions as well and analyze how these interact with those of the customer to influence their mutual experience. The tool allows either real-time or offline feedback to help people reflect on what these interactions mean and figure out how to genuinely elicit better experiences, such as true customer delight. The first deployment of this project focuses on eliciting and capturing smiles, and doing so in a way that is respectful of both customer and employee feelings. This project will also explore ways to share this information and link it to outcomes such as banking fee reductions or donations to charity.

Persons with autism often report extreme hypersensitivity to sound. Researchers believe this hypersensitivity may be related to the acoustic quality of the sound (e.g., its frequency, intensity, and duration), and the context within which it occurs. Our primary aim is to offer persons with autism more control over their acoustic environment, regardless of the context. We are developing new technology for autistic individuals to help them manage their experiences with auditory hypersensitivity. This technology will also offer persons with autism new ways to record and document the sounds they find particularly aversive. Psychophysiological sensors will also be incorporated to assess the role of arousal in auditory hypersensitivity.