Sound Gesture Intelligence

Now we can trigger speech sounds such as syllables, let’s record and segment them in real-time using a button (one of the CoMo.te app for instance). In Babil-on V2, a pair of button-rings have been added to the sensor-gloves allowing for the picking and erasing of voice samples on the fly. Thus, SpokHands and the triggering of pre-made sounds evolved into Hand Sampling, in which the vocal flow is cut and recombined in percussive motor gestures.

The Hand Sampling allows the performer to cut her.his voice in real time, and recombine immediately through gesture. It involves percussive gestures that will segment and trigger vocal fragments. The length of these fragments can vary from syllable to sentence. The order of the re-played segments can be sequential, random, or palindromic, which allows different playing modes. In addition, the quality of the gesture influences the quality of the sound perceived, making the instrument expressive.

Hand Sampling, with Richard Dubelski @ Scene44, Marseille, France, January 2014.

While accelerometric sensors are ideal for applications linked to dynamic motor gestures (beat), as their response time is very fast (on the order of milliseconds), they are unable to estimate static gestures and their positions in space (only angles), and therefore to process symbolic and deictic gestures. This requires the addition of video sensors that can operate in the visible or infrared range.