Zum Hauptinhalt

Sound Gesture Intelligence

Dr. Greg Beller

Direct mapping between voice and gesture

Probabilistic mapping

In this case, the mapping is direct between the detection of a movement pulse or "kick" and the triggering of a sound segment belonging to a buffer, which can be selected:

  • incrementally within the original sequence (to preserve text, for example),
  • inversely or palindromically to the original sequence,
  • probabilistically, by randomly selecting the index of the segment to be played:
    • among all segments each time ([random] in Max)
    • among segments not yet selected ([urn] in Max)
    • among segments according to their probability of appearance ([hist] and [proba] in Max)

In Luna Park, a probabilistic segment/text generation interface has been used (see following figure). The probability distribution (bottom) used to draw the next segment can be manually edited. The initial distribution (middle) corresponds to the frequency histogram of each labeled segment within a buffer (top). In this case, we can speak of a probabilistic mapping paradigm.

edu sharing object

The figure shows a segmented recording in syllabus (on top), a histogram presenting frequency of appearance of each symbol (middle); and the same histogram guided for generation (low), for example by forcing equal appearance of syllables “ni”, “naR” and “noR”.