Research Group Projects and Descriptions

The Music, Mind and Machine group is working towards bridging the gap between the current generation of audio technologies and those that will be needed for future interactive media applications.

Standard loudspeakers transmit sound which necessarily spreads very quickly, and the control of sound projection and position is only about as flexible as where you can hang a loudspeaker. The Audio Spotlight is a device that will project sound much like a spotlight projects light: shining it at a listener allows only that person to hear it, while shining it at a surface causes the sound to appear to originate from there, creating something of a "virtual loudspeaker." Beamsteering will allow the sound to move, enabling the user dynamically to place sound—exactly, and only, where it is desired.

Alumni Contributor(s): F. Joseph Pompei

The automatic classification of marine mammal sounds is very attractive as a means of assessing massive quantities of recorded data, freeing humans and offering rigorous and consistent output. Calculations on a set of vocalizations of Northern Resident killer whales using dynamic time warping have been reported recently. Since this method requires the time-consuming pre-processing measurement of frequency contours, we have explored the use of Gaussian Mixture Models (GMM) and Hidden Markov Models (HMM). These methods can be applied directly to time-frequency decompositions of the recorded signals. Calculations have been made on a set of 75 calls previously classified perceptually into seven call types. With cepstral coefficients as features both HMM’s and GMM’s give over 90% agreement with the perceptual classification, with the HMM over 95% for some cases.

The experience of playing live music over the Internet with musicians located on different continents and audiences at both ends is constrained by network latency. Transmission delays especially effect highly rhythmic music, for instance with Indian percussion instruments, where distributed players are in tight synchrony, either playing simultaneously or participating in a call and response. To overcome time delays we propose an audio system that combines machine listening and music prediction for real-time online musical collaboration.

Musicpainter is a networked, graphical composing environment that encourages sharing and collaboration within the composing process. It provides a social environment where users can gather and learn from each other. The approach is based on sharing and managing music creation in small and large scales. At the small scale, users are encouraged to begin composing by conceiving small musical ideas, such as melodic or rhythmic fragments, all of which are collected and made available to all users as a shared composing resource. The collection provides a dynamic source of composing material that is inspiring and reusable. At the large scale, users can access full compositions that are shared as open projects. Users can listen to and change any piece if they want. The system generates an attribution list on the edited piece and thus allows users to trace how a piece evolves in the environment.

In comparison to our group's Radio-ish project—a one-dimensional music browser—Musicscape is a two-dimensional, spatial music navigation interface designed for browsing large-scale sound archives. It simulates a 2-D sound field by applying Head-Related Transfer Function (HRTF) and enables users to virtually walk around the sound space with a computer mouse or the SpaceNavigator, a 3-D navigation device developed by 3-D connection.

Building on well-developed data visualization techniques, audio is used to enhance understanding by improving the brain's ability to parse information in the scientific process. Sonification (aka auditory display) is the use of non-speech audio to convey information. In collaboration with the Media Lab, NASA is interested in using the human auditory system's powers of organizing and deconstructing sound for the purposes of scientific research and exploratory data analysis. Faced with increasingly voluminous and complex multimodal data and computations, researchers are seeking novel ways to optimize the conversion of information into knowledge. Several sonifications are under study, including spectral data from Mars, sensor data from the Media Lab, space-suit-helmet heads-up displays, new augmentations of the traditional orrery, and enhanced exploration of the Mandelbrot set.

How many decisions does it take before you hear a desired piece of music on your iPod? First, you are asked to pick a genre, then an artist, then an album, and finally a song. The more songs you own, the tougher the choices are. To resolve the issues, we turn the modern music player into an old analog radio tuner, the Radio-ish Media Player. No LCDs, no favorite channels, all you have is a knob that will help you surf through channel after channel accompanied by synthesized noise. Radio-ish is our attempt to revive the lost art of channel surfing in the old analog radio tuner. Let music find you: your ears will tell you if the music is right. This project is not only a retrospective design, but also our reflection on lost simplicity in the process of digitalization.

Musicians and music listeners often describe the quality of musical sounds with words like "bright," "warm," "metallic," or "fat." This project investigates how these verbal descriptors, or perceptual tags, relate to the perception of timbre. We analyzed results from an online survey where more than 1200 participants described the sounds they heard, and found that people use a common terminology to describe timbre, i.e., their choice of words is not linked to their musical or cultural background. We are now looking at how specific words correlate with particular audio features. Our objective is to design an audio-processing engine that can retrieve sounds in a database from a verbal description and that can modify sounds by using words instead of numerical parameters.

Computer games and learning environments increasingly involve humans relating to computational avatars and robots. Intelligence-modeling software requires real-time interaction between the parts, with channel capacities that must match the best of human-human performance. Many multi-modal activities challenge the real-time communication and comprehension speeds between participants. This project aims to enhance human-machine and machine-machine communication capacities between entities in order to encourage new models of interaction.