Y3 | Artificial Intelligence and Computational Tools for Orchestration

Artificial Intelligence and Computational Tools for Orchestration
Y3 | Workgroup Summary

13 July 2021
11:30-13:00 EDT
Via Zoom

Workgroup Leader:
P. Esling (IRCAM)

Aims:

The application and future of AI as it applies to the field of music.

Discussion Points:

Existing models presented

• Computational orchestration (inductive and projective)

• Symbolic music transfer (of high-level attributes)

• Understanding timbre perception (generative timbre space)

• Ultra-light deep learning

• Differentiable digital signal processing

• Model emulating the bow, finger on bow, and hand on pitch.

• Growling model applied to any signing. DDSP in pure data

• Schubert applied the model in a piece, received Ars Electronica 2021

• Flow Synth Universal audio synthesizer control. Learn to control the synth to simplify behavior (giving synth presets from audio files). Presets are organized on a graph.

• Lottery ticket hypothesis: Instruments do not mesh well with deep learning because it is an overwhelmingly demanding process. The lottery ticket hypothesis is the use of a powerful subnetwork with music information retrieval you can filter—making the process more easily applicable in music.

• Neurorack embedded synth. The prototype panel allows communication with other synthesizers (to be developed).

Full Q&A Session:

• Q: With Neurorack: to really get refined expression with strings to get high quality—how long will it take and how do you extend to other instruments?

• A: Need rules for expression when going from score. Training takes instead of fixed; you learn from sinusoidal model. Simple, SMS synthesis works on decomposing anything; however, it is a list of partial. Trains in 2h, can go for 30 min of recording.

• Expressivity: problem is the expressivity shifts to F0 z loudness info—Not pitch or expression. A real violin curve is expressive. When we are changing the curve, the expressivity goes away.

• Need another core ensemble to apply and test the model? Take the tools and roll them out.

• You build it and they will come (is this the method?) how much buy in? (4 projects) feedback loops

• Flucomo at Huddersfield: a series of composers engaged tools and their various applications. This group of individuals try to expand the possibilities of these tools.

• Misses and glitches, are they interesting? Often, composers want to be surprised. Constructivist and understandable. Pro-technology OR is against ai with the fear to be “a slave to the tool” this sort of innovation and application is more welcome in popular audio, the film and game industry; however, note widely used in contemporary compositions.

• The models can be personalized—transfer violin to make cat noises. The starting point is from science and not from arts. (And this is a gap worth closing.)

• Should AI be calibrated for the public AND/OR malleable for composers? Composers would like to control on techniques and process to make them individualized, malleable, and tweakable. They are attracted to something adaptable to different contexts. All models can be dedicated to tasks (personalized to composer, sounds, and interactions) Composers also often like, break, crash and errors. They are often interested with what is not supposed to be working.

• Back and forth between science and creative arts is needed to make these tools more applicable. The composer’s requests would give challenges to the researchers.

• Raster-Norton collaborated for 2 months to build a model from their discography (built 4 songs) generated tracks by selecting a start and end. AI builds the rest from there.

• (DDSP) the model will not work for Extended contemporary techniques.

Full future work ideas:

1.     Model with a more complex construct of space in middle (a carillon)

2.     data set of wolves—pending

3.     learn from performances. (Symbolic scores) fine pitch and loudness curves containing vibration and attack details (with master student—in 2 months) structures in the symbolic s put symbolic score—result a very nice sound).

4.     goal is to make one model per instrument family. It works for the same family (i.e., strings).

5.     make a call with demo videos sent out to the actor newsletter to invite performers and composers to start using AI and to collaborate as a group! (Fall 2021)

6.     organize easy user methods to invite users. Usability from amateurs will allow them to go further and to challenge the model for further innovations.

7.     Documentation is necessary. In 6 months: List of models, information regrouped, Simple database, and how to use them. In 1 year, user videos, links (tutorials?) (The works.)

8.     Composer Wishlist. Plan a call for participation in early fall to connect with performers and composers who can start to challenge the model by providing a list of wishes. The challenge will help the model be more useful while providing compositional tools.

9.     (Network bending) explore unnatural playing modes.

Action Items:

1.     Reduce the “isolation” of AI group within ACTOR.

a.     Apply to the research project call.

b.     Entice more collaborative work within the project.

2.     Increase the usability of AI tools amongst composers

a.     Create a list of available models and their use (grouped information, database)

b.     Create an extensive documentation for each model with links.

c.     Perform some demo videos, tutorials and other use cases.

3.     Perform a “challenge request” for composers

a.     Composer Wishlist (2-way communication).

b.     Plan a call for participation in early fall to connect with performers and composers who can start to challenge the model by providing a list of wishes. The challenge will help the model be more useful while providing compositional tools.

4.     Work on high-quality performance rendering

a.     Evaluate the new violin-rendering model.

b.     Goal is to make one model per instrument family (works for the same family).

Follow-Up:

• DDSP, need actual curves pitch, loudness. Sound natural. (Find pitch and loudness that hit certain scores Producing ultra-high-quality rendering)

• Perhaps play the DDSP violin with hand (instead of iPhone and Midi Keyboard)

• Get recording of violin hand position and mirror the position for use.

• Eurorack for all instruments. Need to do this association (score to performance before performance to audio)

• Find participants for a research creation project!

• rendermen. Rendering engine faster than real time. 20k presets. In 4:40 a.m. h of playing time.

• For Co-improvisation to the timbre world. Statistical model of relationships must be built. It is difficult to abstract pitch and harmony-input to isolate timbre.