Professional Documents
Culture Documents
Dicy2 For Max
Dicy2 For Max
Version 3.0
2 Dicy2 Concepts 5
2.1 Core Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.1.1 The Server . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.1.2 The Agent . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2 Memory and Scenario . . . . . . . . . . . . . . . . . . . . . . . 7
2.2.1 Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2.2 Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.3 Compositional approaches . . . . . . . . . . . . . . . . . . . . 9
2.3.1 Offline . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.3.2 Real-time . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3 Audio Workflow 12
5 Further Exploration 26
1
6.1 Some references . . . . . . . . . . . . . . . . . . . . . . . . . . 27
6.2 Authors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
6.3 Artistic collaborations . . . . . . . . . . . . . . . . . . . . . . . 27
6.3.1 An instrument designed through artistic productions . 27
6.3.2 Example and tutorial files . . . . . . . . . . . . . . . . . 28
6.4 More . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2
1 Dicy2 library for Max
1.1 Introducing Dicy2
Dicy2 for Max is a suite of Max abstractions which use machine learning
for interactive generation of musical sequences. It can be integrated into
musical situations ranging from the production of structured material within
a compositional process to the design of autonomous agents for improvised
interaction. It is available as a plugin for Ableton Live and a library for Max.
There is a discussion group for the Dicy2 library for Max, which can be
found at the Ircam Forum: https://discussion.forum.ircam.fr/c/dicy2/.
• Install the dependency "MuBu for Max" version 1.10.5 or higher from the
Ircam Forum Webpage https://forum.ircam.fr/projects/releases/mubu/.
RELEASE NOTE The current release of Dicy2 has been tested with MuBu
v1.10.5 which can still be found at https://forum.ircam.fr/projects/releases/mubu/
in case of hypothetical problems with a later version. Dicy2 will of course
be tested and updated along with the successive MuBu releases.
1.2.2 Installation
/patchers:
3
• Two core abstractions: dicy2.server.maxpat and dicy2.agent.maxpat.
/examples:
/externals: These max externals are used internally by some of the Dicy2
abstractions.
• spat5.shell
/misc:
• Dicy2_server.app
The Dicy2_server.app is the background app which does the heavy lifting
of Dicy2, and communicates wih the Dicy2.server for its communication with
the Max environment.
/media:
/extras:
Warning The first time you use Dicy2, you will probably get a pop-up mes-
sage asking you to allow dicy2.server.app to run. Click OK.
4
2 Dicy2 Concepts
Dicy2 uses machine learning to generate structured variety from musical
source material. This basis material is used to train the dicy2.agent object,
which uses machine learning techniques to create an internal map of temporal
relationships within the basis material. The user then sends queries to the
agent (called a Scenario), and Dicy2 returns segments of musical material
in response. Using pattern recognition algorithms, it looks for a ‘best match’
between the Scenario and the material stored in the memory, taking into
account both musical similarities and the temporal structure of the Memory.
Once the Memory is built, you can send Dicy2 sequences of these labels
(called a Scenario), and it will return optimized content that constitues the
’best match’ for the label sequence. The content (which is output as a list)
can then be sent to further patches for rendering. The illustration below
presents a schematic of the Dicy2 workflow, showing the most common
helper abstractions.
5
2.1 Core Objects
2.1.1 The Server
Sending the dicy2.server the ’initialize’ message will launch the back-
ground app, and open up OSC ports for communication with the background
app and the Agents. The server and the background app can be stopped by
sending the server the ’exit’ message.
Also very important: you should always have a [freebang] object send an
’exit’ message to the Server in order to shut down the background app.
The dicy2.agent abstraction is where you will have most of your interaction
with the Dicy2 environment. You will train the Agent on a memory, and send
Scenarios to the Agent, which will generate optimized responses of musical
6
material. You may have any number of Agents operating simultaneously, all
communicating with the same Server. Each Agent requires a unique name
as an argument. It expects a Scenario in its left inlet, and returns a content
sequence from its left outlet.
If your content is limited, you can write it by hand and store it in any kind
of text file. For example, here is part of the word-based memory from the
server_and_agent tab in tutorial 0, which is stored in a coll object:
Here, the first item in each line is a part-of-speech label, and the second the
7
content.(Note that in line 31, "am driving" is in quotes to make it a single
symbol.) In this context, a ’dump’ message to the coll will send out one line
at a time. Prepending each line with the ’learn’ message will train the Agent.
2.2.2 Scenarios
• pick: This message, used in place of a label, gives the Agent a free
choice for that member of the sequence. This is not a random choice,
but one that will be optimized within the context of the scenario overall.
• free: the message ’free {int}’ is the equivalent of sending {int} number
of pick messages, giving the agent a wholly free choice of optimized
sequences.
8
Figure 4: How an Agent responds to a Scenario
When presenting scenarios to the Agent, there are two general overall ap-
proaches, offline and real-time. In an offline scenario, a structured sequence
of labels is presented to the agent, and the returned content can be con-
sidered as a musical gesture, section, or even an entire composition. (See
tutorial 00, offline_scenarios).
9
start date.
2.3.2 Real-time
10
in the history which was last played. In interactive applications, you can
synchronize the clock as each event in the current response is rendered using
the ’step’ message, which increments the internal time pointer by 1. This
guarantees that a new scenario will be correctly optimized. (See tutorial 0,
real_time_scenarios.)
11
3 Audio Workflow
One of the most common use cases for Dicy2 is one in which segments from
a prerecorded audio file is used as accompaniment to a live performer. In this
situation, the audio file is segmented and analyzed using one or more audio
descriptors, such as pitch, noisiness, etc., and each segment is labeled based
on those descriptors. The Agent is then trained on the soundfile, creating
a memory model based on the temporal sequence of the labels. Real-time
audio, from either a live or prerecorded source, is then played into the system,
and it is analyzed using the same descriptors as were used to define the
memory, and labels are output corresponding to the labels in the original
prerecorded file. These labels are then used to generate Scenarios, which in
turn drive the Dicy2 agent to play back matching segments, optimized to the
temporal structure of the original file.
The above illustration shows a typical Dicy2 audio setup. One can think
12
of the workflow as follows:
13
4.1 Using the Memory Maker
The dicy2.easy_memory-maker segments, classifies, and labels an audiofile,
and saves this data for later use. It takes one mandatory argument: a Memory
name.
14
Firstly, one loads an audiofile into the memory-maker, either by dragging
a soundfile to the topmost pane, or using the ’read’ button. Once this is done,
the file is ready to be analyzed.
In the ’Audio segmentation and analysis’ pane, you must select at least two
descriptors: one to determine segmentation (which is typically Loudness,
but for extremely legato material, other options may work better), and then
one or more additional descriptors, which will determine the labeling of the
segments. If you are looking to match melodic and harmonic color, the best
starting point for analysis is Chroma. To reinforce the melodic aspect, add
PitchClass, and to emphasize the timbre, add HarmonicSpectralCentroid.
What this means in the context of Dicy2 is that there is a tradeoff between
15
the amount of variety the agent can generate and the number of classes.
With a large class count, each class will have more individual character,
but as the agent will have fewer choices in that class, the results will be
more predictable. If there are fewer classes, the temporal model will have
a greater number of options when responding to a scenario, but with too
few there is less of a sense of distinction between classes. For the typical
analysis mentioned above, Chroma alone works best with 20 classes, and
Chroma+HarmonicSpectralCentroid with 30.
Once the segmentation module has sliced the soundfile into segments,
and assigned each segment a set of descriptor values, this module looks at
the descriptor values for each segment, and groups groups segments based
on their relative distance in a data space whose dimensions are determined
by the number of analysis descriptors, assigning each class a numerical label.
These are the labels the dicy model will use when receiving and responding
to queries.
16
This is often an iterative process, and once you have completed your
analysis, it is worthwhile to examine your classes, which you can do by clicking
on the ’inspect classes’ button, which will open up the Class Inspector:
On the left is a jit.cellblock, listing the class labels, and the number of
segments in each class – you will notice some classes are empty. The
multislider on the right is a histogram which displays visually the number
of members in each class. In the bottom pane one can listen to them by
selecting the label, and choosing from the ’Segment index’ menu. Listening to
the classes can give you a sense of how much similarity and variation there
is amongst the members of each class. You can also view the classification
in a scatterplot:
17
Figure 12: classification scatterplot
Once you have tuned your classification and segmentation to your liking,
you can train the the Agent to classification data to with the ’dump’ button.
(What the Agent actually learns is a sequence of label markerID pairs, de-
signed for eventual playback by a mubu.concat object.) Finally, you can write
the Memory to disk. When you do this, you will get two save dialogs in order.
That is because an audio-based memory is comprised of three separate
documents:
It is usually best practice to give all of these files the same name, with their
respective extensions, as the dicy2.easy_loader utility is designed to handle
identically named files.
18
Finally, you will notice that there is a field labeled ’memory name’ in the
bottom right corner. This is an argument which must be shared by all other
easy abstractions which will be using this Memory. (It need not be the same
as any Memory you create, as any Memory can be loaded into they system)
In practice, this means:
• dyci2.easy_memory-maker
• dyci2.easy_analyzer-identifier
• dyci2.easy_loader
• dyci2.easy_sequencer-renderer
19
4.2.1 Segmentation and Analysis
4.2.2 Gating
The analysis timing modes determine the rate and method by which the
analyzer-identifier identifies and labels incoming audio. There are three pos-
sible modes:
• Instant mode: the analyzer looks at the incoming audio stream and
analyzes a short time window to estimate the likely timing and label of
the next event. The size of this analysis is set by the ’next window size’
parameter.
20
4.3 Using the scenario-maker
• repeat label: Just like it says: generates scenario length copies of the
label.
• start with label then pick: The module will generate a sequence be-
ginning with the last received label, and allow the agent to create an
optimized sequence from there.
• Mix label and pick: randomly selects between the last label and the ’pick’
message, which allows the agent to choose the optimized decision.
The ’% chance’ number box determines what percentages of labels will
be chosen.
21
• last n labels: Presents the last n labels received from the analyzer. the
number of labels is determined by the ’scenario length’ parameter.
• mix last n labels and pick: randomly selects between the list of last n
labels and the ’pick’ message. The ’% chance’ number box determines
what percentages of labels will be chosen.
• scenario length: the number of items sent out in response to a label (or
bang) received by the scenario-maker.
• chance of ’label’: the percentage chance that a given item will pass a
label or a pick message. At 100%, no pick messages will be passed. This
parameter only effects generation methods 1, 2, and 4 (zero-based).
22
• Right: sends a ’step’ message after each event is played, designed to
be connected to the Agent’s internal clock pointer.
4.4.1 Sequencer
23
4.4.2 fadein/out
These two values determine the fade in and out times of the entire sequence
the sequencer-renderer has received. the fade in is timed in milliseconds,
and the fade out by a number of events, so you can fade out over a certain
number of events without having to calculate the duration of those events.
4.4.3 Renderer
This section exposes a few of the many options available to the mubu.concat ob-
ject.
24
As we have mentioned, an audio-based Memory is comprised of three docu-
ments:
• A .mubu file containing analysis and segmentation data for the audio
file.
The easiest way to work with the easy_loader is to save all three documents
with the same name, for example:
• SomeMemory.wav
• SomeMemory.mubu
• SomeMemory.json
The loader uses the audiofile (.wav, .aif, or .aiff) as the key to all three files.
Dragging the audio file portion of the memory, or opening it with the ’load’
button, will load the audio file, plus the associated MuBu and dictionary files,
and automatically dump the analysis data to the Agent.
4.6 Behaviours
Since it it possible to load different Memories into the same set of audio
objects on the fly, It can often be convenient to use remote messages to the
various easy abstractions to wrap all the parameters into a single package –
this is called a Behaviour. There is no official format for Behaviours, it is just
a series of messages.
25
Figure 17: A typical Behaviour
5 Further Exploration
Although this document has concentrated on real-time audio interactions, the
basic Dicy2 workflow of developing a memory and passing scenarios can be
applied to almost any set of temporally structured data, such as MIDI and text
files, some instances of which you can find in the example patchers. You can
also find more in-depth coverage of developing scenarios offline, as well as
examples of using Agents to control other Agents, sharing Memories between
Agents, and much more. We encourage you to examine the examples and
the performance strategies carefully, as they can provide a springboard for
further work of your own based on the Dicy2 system.
26
6 More about Dicy2
6.1 Some references
If using Dicy2, please quote Nika, J., Muller, A., Borg, J., Ostrowski, M., &
Assayag, G. (2022, December). Dicy2 for Max. Ircam. 2022 or
Nika, J., Déguernel, K., Chemla, A., Vincent, E., & Assayag, G. (2017, October).
Dyci2 agents: merging the" free"," reactive", and" scenario-based" music
generation paradigms. In International Computer Music Conference (1).
Additional article references are given in the "References" section (2; 3; 4; 5; 6).
6.2 Authors
Dicy2 is a library for Max and a plugin for Max for Live of the Ircam Musical
Representations team, designed and developed by Jérôme Nika, Augustin
Muller (Max library), Joakim Borg (Python generative engine / Gig RepMus
API), and Matthew Ostrowski (tutorial patchers and videos, abstractions) in
the framework of the projects ANR-DYCI2, ANR-MERCI, ERC-REACH directed
by Gérard Assayag, and the UPI-CompAI Ircam project.
The audio use cases have been designed and developed with Diemo
Schwarz and Riccardo Borghesi, and use the MuBu(7) and CatArt(8) envi-
ronments of the ISMM team of Ircam. Max4Live plugin by Manuel Poletti.
Contributions / thanks: Serge Lemouton, Jean Bresson, Thibaut Carpentier,
Georges Bloch, Mikhaïl Malt, Axel Chemla–Romeu-Santos, Vincent Cusson,
Tommy Davis, Dionysios Papanicolaou, Greg Beller, Markus Noisternig.
27
National des Arts Contemporains, Vir Andres Hera, Gaëtan Robillard, Benoît
Delbecq, Jozef Dumoulin, Ashley Slater, Hervé Sellin, Rodolphe Burger, Marta
Gentilucci... After having evolved research prototypes crystallizing the con-
tributions of these various projects for several years, a collaborative work
carried out during the year 2022 has led to the finalization of a release of
Dicy2 as a plugin for Ableton Live and a library for Max.
This distribution includes agents and sound files from past productions with
our friends and collaborating musicians and composers who helped bring
Dicy2 to life (courtesy of the artists). Please do not use these agents and files
in any context other than these tutorials to respect their work and generosity.
List of files
• – Doublebass_Perrot_Fox.wav,
– Guitar_Caillou_Fox.wav,
– and Voice_Daumergue_Fox.wav
were respectively recorded by Alex Perrot, Thomas Caillou, and Manu
Daumergue during Rémi Fox’s residency at Ircam for the concerts and
first album of "C’est pour ça".
• – Balafon_Lehman_ExMachina.wav,
– and SaxPlayingMode_Lehman_ExMachina.wav
were recorded by Steve Lehman for "Ex Machina" with Orchestre Na-
tional de Jazz.
28
6.4 More
• Please write to [email protected] and [email protected]
for any question, or to share with us your projects using Dicy2!
• Interested developers can check out the generative core of Dicy2, im-
plemented as a Python library.
• License: GPL3
References
[1] J. Nika, K. Déguernel, A. Chemla, E. Vincent, G. Assayag, et al., “Dyci2
agents: merging the" free"," reactive", and" scenario-based" music genera-
tion paradigms,” in International computer music conference, 2017.
29