Zum Hauptinhalt

Sound Gesture Intelligence

Dr. Greg Beller

Introduction

The Synekine project

By analogy to synesthesia, a phenomenon by which two or more senses of perception are associated, the neologism “synekinesia” would reflect our ability to associate two or more motor senses. In the Synekine project, the natural link between voice and gesture is enhanced by "creative prostheses". Technology allows the transformation of the voice by capturing the gesture. The voice is conventionally captured by a microphone, while the positions and dynamics of the hands are informed by different motion capture sensors.

Between the two, different computer programs based on artificial intelligence allow the direct manipulation of sound with the hands. The performer can record and arrange her or his voice in space, sample and play vocal percussion, and design a complete sound stage. This technology makes it possible to establish a link between voice and gesture, between sound and movement, between time and space. This link will be referred to as mapping in the rest of this unit. The following table gives a chronological overview of the work carried out as part of the Synekine project, the different instruments developed and a list of the various technical devices used.

edu sharing object

Table: Chronology of the works realized within the framework of the synekine project presenting the various instruments elaborated from various technical devices.


Linking space and time through movement then transforms the search for sound into a scenic exploration. These instruments have been used to create performances and installations at the crossroads of theater, dance and music.