Interactive Machine Learning for Music

IML as described above can be an ideal approach to building instruments and other real-time interactions under certain circumstances:

• You understand the behaviour you want the model to take on, and you are capable of providing training examples that illustrate that behaviour, evaluating whether a model is satisfactory, and identifying model mistakes.
• The representation of the input data (e.g., numbers used to represent sensor values) does not have too complicated a relationship with the output. (Of course, “too complicated” is hard to define, but something like feeding in a raw audio waveform as an input will certainly be too complicated. In general, the more complex this relationship, and the greater the number of input dimensions, the more training examples may be needed and the more sophisticated a learning algorithm may be needed.)

In these circumstances, IML allows you to craft training sets that accurately communicate your intention, and you are able to build suitable models from a relatively small number of examples (e.g., dozens, or maybe hundreds of examples) without taking too much time to source examples or wait for training.

On the other hand, IML may not be a good fit when:

• You already have a training set which you are confident represents the patterns between inputs and outputs very well, and this is unlikely to be improved.
• You want to build a model that accommodates a diverse variety of users, e.g. doing accurate gesture recognition on a wide variety of body types. (In this case, using training data only from yourself may give you a model that fails unexpectedly for other users, and you should probably spend time collecting a substantial dataset from diverse users.)
• You suspect you may have a more complicated relationship between inputs and outputs than can be supported by simple (fast) learning algorithms applied to small datasets (e.g., because you’ve tried IML and it hasn’t worked).

In these cases, you may be better off using a more conventional approach to ML, with tooling (e.g., Python libraries and offline experimentation) that focuses your effort on finding the best “feature representation” and learning algorithm configuration for a particular dataset.