Research Group Projects and Descriptions

The Opera of the Future group (also known as Hyperinstruments) explores concepts and techniques to help advance the future of musical composition, performance, learning, and expression. Through the design of new interfaces for both professional virtuosi and amateur music-lovers, the development of new techniques for interpreting and mapping expressive gesture, and the application of these technologies to innovative compositions and experiences, we seek to enhance music as a performance art, and to develop its transformative power as counterpoint to our everyday lives. The scope of our research includes musical instrument design, concepts for new performance spaces, interactive touring and permanent installations, and "music toys." It ranges from extensions of traditional forms to radical departures, such as the Brain Opera and Toy Symphony.

"Death and the Powers" is a groundbreaking opera that brings a variety of technological, conceptual, and aesthetic innovations to the theatrical world. Created by Tod Machover (composer), Diane Paulus (director), and Alex McDowell (production designer), the opera uses the techniques of tomorrow to address age-old human concerns of life and legacy. The unique performance environment, including autonomous robots, expressive scenery, new Hyperinstruments, and human actors, will blur the line between animate and inanimate. The opera will open in Monte-Carlo in fall 2010 and then tour worldwide.

Early in the opera "Death and the Powers," the main character, Simon Powers, is subsumed into a technological environment of his own creation. The theatrical set comes alive through robotic, visual, and sonic elements that allow the actor to extend his range and influence across the stage in unique and dynamic ways. This environment must compellingly assume the behavior and expression of the absent Simon. In order to distill the essence of this character, we recover performance parameters in real time from physiological sensors, voice, and vision systems. These gesture and performance parameters are then mapped to a visual language that allows the off-stage actor to express emotion and interact with others on stage. Additionally, we use these gestural performance parameters for vocal manipulation. Our approach takes a new direction in augmented performance by employing a non-representational abstraction of a human presence that fully translates a character into an environment.

The Hyperinstrument project creates expanded musical instruments and uses technology to give extra power and finesse to virtuosic performers. They were designed to augment a wide range of traditional musical instruments and have been used by some of the world's foremost performers (Yo-Yo Ma, the Los Angeles Philharmonic, Peter Gabriel, and Penn & Teller). Research focuses on designing computer systems that measure and interpret human expression and feeling, exploring appropriate modalities and content of interactive art and entertainment environments, and building sophisticated interactive musical instruments for non-professional musicians, students, music lovers, and the general public. Recent projects involve both new hyperinstruments for children and amateurs, and high-end hyperinstruments capable of expanding and transforming a symphony orchestra or an entire opera stage.

Alumni Contributor(s): Roberto M. Aimi, Mary Farbood, Ed Hammond, Tristan Jehan, Margaret Orth, Dan Overholt, Egon Pasztor, Joshua Strickon, Gili Weinberg and Diana Young

Hyperscore is an application to introduce children and non-musicians to musical composition and creativity in an intuitive and dynamic way. The "narrative" of a composition is expressed as a line-gesture, and the texture and shape of this line are analyzed to derive a pattern of tension-release, simplicity-complexity, and variable harmonization. The child creates or selects individual musical fragments in the form of chords or melodic motives, and layers them onto the narrative-line with expressive brushstokes. The Hyperscore system automatically realizes a full composition from a graphical representation, allowing individuals with no musical training to create professional pieces. Currently, Hyperscore uses a mouse-based interface; the final version will support freehand drawing, and integration with the Music Shapers and Beatbugs to provide a rich array of tactile tools for manipulation of the graphical score.

Alumni Contributor(s): Mary Farbood, Ed Hammond, Tristan Jehan, Margaret Orth, Dan Overholt, Egon Pasztor, Joshua Strickon, Gili Weinberg and Diana Young

There have been few studies of bowing data from real players, in part due to the difficulty in capturing this information. We have designed an interface to measure bowing parameters produced by real players, while maintaining the portability and playability of a traditional violin bow. This interface, the Hyperbow, consists of a carbon-fiber violin bow with a custom wireless sensing system (with accelerometers, gyroscopes, electric field position and force sensors). The Hyperbow is partnered with a Yamaha Silent Violin SV-200 also augmented with gesture sensors. This playable measurement system is the core component of an experimental setup used to investigate the bowing parameters produced by real violinists. In these investigations, the parameters are recorded with the sound produced during performances of different bowing techniques (détaché, martelé, spiccato). These data can then be analyzed to help understand the various strategies employed by violinists to achieve similar goals in sound production.

Alumni Contributor(s): Diana Young

The scientific community is making marked progress in the area of Alzheimer's disease (AD) treatment: memory-related pharmaceuticals are available, the neurobiology of AD is fairly well understood, and the genetic underpinnings of the disease continue to be unraveled. However, despite these advances, it has been shown that individuals often present the symptoms of AD years before they seek a diagnosis. The barrier to treatment is the lack of structure with which to obtain a diagnosis or even predict the onset of disease in a stigmatized environment. With technology, we can build clinically valid assessment into the tools we use every day—the tools we care about. We are developing music tools to detect cognitive performance in the memory domains at risk of decline in the earliest stages of AD. These tools are mobile, longitudinal, and the patient is the first point of feedback.

The Hyperinstruments group is exploring new ways to visualize music both at the song-level and the collection-level. At song-level: SoundSieve is a music visualizer that takes the intrinsic qualities of a musical piece—pitch, time, and timbre—and makes their patterns readily apparent in a visual manner. For example, you can quickly pick out repeating themes, chords, and complexity from the pictures and video. At collection-level: MusicBox focuses on the problem of navigating a large body of music. It aims to help you find music you like, both inside your own collection (to match a particular mood, for example), or from a body of entirely new music. MusicBox visualizes your music collection in space, giving each track a location based on how similar it sounds to other tracks. This new manner of navigation stands in stark contrast to traditional, text-dependent media players like iTunes and Windows Media Player.

Our work in Music, Mind, and Health continues to push forward, showing the technologies and perspectives required to build on the transformative nature of music to drive specific neurological, physical, emotional, and psychological change in the clinical setting and for the general public. A radically new "Personal Instrument" is currently being used by Dan Ellsey, a quadraplegic individual, who controls this interface to sculpt an expressive performance of music in real time. A three-month study of Ellsey's expressive behavior—its potential as well as its limits—resulted in an interface tailored just for him, enabling him to access expressive performance despite his physical disability. This new line of work highlights principles for future instruments and applications, where the impact is in the marriage of the interface and uniqueness of the person. In this way, we are pursuing new design philosophies, technologies, and collaborations within the scientific community, public performance, and clinical research.

Robots and performers make beautiful music together. The opera "Death and the Powers" will feature a chorus of seven-foot tall, autonomous, polymorphic robots which at various times function as characters, set pieces, and lighting elements. Using state-of-the-art control electronics and a novel real-time performance recovery protocol, a total of 16 individually addressable BobBots will reflect on, participate in, and illuminate the action onstage.

Alumni Contributor(s): Mike Fabio

MusicBox focuses on the problem of navigating a large body of music. It aims to help you find music you like, both inside your own collection (to match a particular mood, for example), or from a body of entirely new music. MusicBox visualizes your music collection in space, giving each track a location based on how similar it sounds to other tracks. This new manner of navigation stands in stark contrast to traditional, text-dependent media players like iTunes and Windows Media Player.

Personal Opera is a radically innovative creative environment that enables anyone to create musical masterpieces sharing one’s deepest thoughts, feelings, and memories. Based on our design of, and experience with, such projects as Hyperscore and the Brain Opera, we are developing a totally new environment to allow the incorporation of personal stories, images, and both original and well-loved music and sounds. This development is based on two guiding principles: first, that active music creation yields far more powerful benefits than passive listening; and second, that increasing customization of the musical experience is both desirable and possible, as evidenced in our group’s development of Personal Instruments (see Music, Mind, and Health) and Personal Music. Personal Opera goes a step further, using music as the medium for assembling and conveying our own individual legacies, representing a new form of archiving, easy to use and powerful to experience.

Skellig is an opera with music by Tod Machover and a libretto based on the best-selling novel for young people by David Almond. It premiered in the UK in November 2008. Besides blending acoustics and electronics, natural noise, and soaring melodies, Skellig also presents several live performance breakthroughs. A non-professional teenage chorus is used throughout, blended seamlessly with high-level professionals; this chorus is guided by an interactive "sonic score" that provides auditory cues, textures to imitate, and electronic reinforcement for the entire 100-minute show. In addition, specially designed "ambisonics" were developed to allow sound to emanate from the stage and engulf the audience in all dimensions, the first time such a technique has been used in a full-scale theatrical setting.

SoundSieve is an exciting new way to look at the structure of music, designed to create a visual “fingerprint that can provide the listener with instant information about any given piece. SoundSieve aims to counter the trend towards overly simplistic, flashy music visualization programs by creating a real-time visualization that is not only informative, but enlightening as to the underlying musical structure of the piece. From any MP3 file, SoundSieve uses simple, intuitive mapping to create a picture and video playback that highlight key audio features. In addition, SoundSieve allows the user to actively examine the piece: to zoom in on portions of interest, adjust playback, and add informative overlays. SoundSieve has recently been used to create customized animations that highlight particular features in a piece (e.g., Tod Machover’s "The System" for the opera "Death and the Powers").

The Chandelier is a large-scale robotic musical instrument that is being developed for "Death and the Powers." Its 48 strings can be actuated both through powerful electromagnets, and tactilely (plucked like a harp or bowed like a cello). With the strings driven by electromagnets, the tactile player can also repeatedly damp strings or create overtones by carefully touching the strings' anti-nodes, creating a new intimacy between players, who play not just the same instrument, but the same strings. The Chandelier is composed of many systems—logic for control of music and lighting, networked servers, and playable interfaces—all built around an elegant, articulated skeletal structure that allows changes to the length, angle, and tensions of the strings. We are currently experimenting with playing it through new types of interfaces to take advantage of its unusual tuning and sonorities.

Toy Symphony combines children, virtuosic soloists, composers, and symphony orchestras around the world to alter radically how children are introduced to music, as well as to redefine the relationship between professional musicians and young people. A complete set of Music Toys will be distributed to children in each host city (including Berlin, Dublin, Glasgow, Manchester/London, and Tokyo), where children will be mentored to create their own sounds and compositions for toys and traditional instruments. A pedagogy for using these Music Toys to teach and to instill a love for musical creativity will also be developed. Final concerts will be presented in each host city including children's compositions and specially commissioned works by young composers, to be performed by children, soloists, and orchestra, playing Music Toys, Hyperinstruments, and traditional instruments.

Alumni Contributor(s): Roberto M. Aimi, Mary Farbood, Tristan Jehan, Ariane Martins, Laird Nolan, Gili Weinberg and Diana Young

The Trainer Piano actively moves your fingers as you learn how to play. Employing active magnetic force below the keys of a working piano, the Trainer Piano provides users with kinesthetic input that augments their normal motor learning process. By providing a "feel" for what a user is supposed to play, the Trainer Piano minimizes the amount of time necessary to learn new motor patterns and acquire new motor skills. The core idea behind this project lies in the hypothesis that computer-controlled force can be used to teach students how to play an instrument at a faster and more efficient rate than would occur in an unaided learning environment.

The Vocal Augmentation and Manipulation Prosthesis (VAMP) is a gesture-based, wearable controller for live-time vocal performance. This controller allows a singer to capture and manipulate single notes that she sings, using a gestural vocabulary developed from that of choral conducting. By drawing from a familiar gestural vocabulary, this controller and the associated mappings can be more intuitive to both performer and audience. This instrument will be used in the upcoming opera Death and the Powers by the character of Nicholas.