Three-Dimensional Score: Seeing Music, Hearing Sculpture

Year: 2016 Authors: Miika Karttunen; Atte Tenkanen

Core claim

A 3D vector-based mapping of tonal music can turn MIDI data into analyzable and 3D-printable virtual sculpture.

Topics

music visualization, tonality mapping, 3D sculpture, MIDI data

Domains

modulo 12 arithmetic, multidimensional scaling, kernel density estimation, vector geometry, sculpture, data art, 3D printing, visual music

Methods

Vector Sequence Model, Krumhansl-Schmuckler key-finding algorithm, pitch-class segmentation, kernel density estimation

Media

MIDI, virtual 3D objects, STL, PLY

Paper text

The text below is the locally extracted OCR/Markdown version of the paper. Raw PDF files remain local and are not published here.

Three-Dimensional Score: Seeing Music, Hearing Sculpture

Miika Karttunen* Department of Mathematics and Statistics University of Turku 20014 Turun yliopisto, Finland mlkart@utu.fi

Atte Tenkanen Department of Musicology University of Turku 20014 Turun yliopisto, Finland attenka@utu.fi

Abstract

We are looking for the shape of music, to make that invisible art visible—even palpable. Vector Sequence Model (VSM) is a procedure that maps musical MIDI data into virtual sculptural form. Its analytical strength is based on the tonality detection of music. For the purpose, we have developed a three-dimensional version of the traditional key circle of fifths, major-minor key ball. With VSM, musical time is associated with vectors and their movements in three-dimensional space.

Overview

The representation of data using sound is called data sonification. With sonification, non-musical data of whatever type are converted into musical data. Similarly, several models have been proposed for mapping musical data onto architectural and sculptural data. The most important motives have been artistic and experimental interests (see, for example, [1]).

The basic difference between music and sculpture has to do with time. In order to solve the mapping problem with respect to these two art forms, we must first answer several related questions as: 1) How can musical time be mapped in three-dimensional space? 2) How can concave sculptural forms be derived from linearly and inexorably progressing music?

When listening to music, we hear it one moment at a time. Soon that moment is gone forever, since time is linear (in this scale) and never returns to its starting point. In a two-dimensional as well as a three-dimensional coordinate system, time can be represented as one dimension. If we want to map the temporal quality of music in three-dimensional space and—at the same time—produce concave angles in any possible direction, we cannot tie the time of music to any single dimension. We solve this dilemma by deploying vectors.

Vector Sequence Model

With the Vector Sequence Model (VSM), our most important aim is to develop a mapping procedure in which the connection between musical tonality and a sculptural form can easily be seen. With VSM, musical time is associated with vectors and their movements in three-dimensional space. For this application, we define music as a set of notes that can be expressed in MIDI format: pitches are expressed as equal tempered MIDI pitches, which in turn can be transformed into 12 octave-equivalent pitch classes using modulo 12-arithmetic. The twelve pitch classes are equal to note names (C, C#…B) inside an octave domain and are usually marked with the numbers 0–11. Temporal information is based on note-onset times and note durations in MIDI ticks. The concept of sculpture consists here of virtual 3D objects. We denote the mapping of the VSM procedure $f$ as follows:

$f:Mu\to Ma\to Sc,$

Karttunen and Tenkanen

where $M_a$ includes music analytical methods and other mathematical procedures that refine the original musical data.

Instead of grounding the model in basic low-level musical parameters, such as pitch and duration, as is the case, for instance, with the works of Hansen [1], VSM is based on an upper level parameter, tonality. ${}^1$ Tonal tensions, especially larger-scale key modulations, often play an important role in music perception. With VSM, such a key modulation is associated with the change of direction in 3D space. Major-minor tonalities are extracted from musical data by segmenting pitch data into pitch-class sets with the same size (cardinality)[7] and defining their tonalities using Krumhansl-Schmuckler key-finding algorithm [4]. A tonal key is defined separately for each bar. A bar, for its part, is associated with a unit vector in 3D space and its direction is determined based on the major or minor key. The directions of the keys in 3D space are determined according to a major-minor key ball (Figure 1b), which is in a way an expanded version of the traditional major-minor key circle of fifths (Figure 1a). The directions are calculated by applying multidimensional (here: three-dimensional) scaling (MDS)[5] to all 24 (major and minor) Krumhansl-Kessler key profiles [6]. We have attached colors from the red-yellow-blue (RYB) circle onto the key circle of fifths to make the final visual impression more analytical (Figure 1a). The farthest keys—a tritone apart—are colored with the opposite colors.

Figure 1: a) RYB-colored key circle of fifths used in the current project. The major keys are written in capitals and the minor keys with small letters; b) Major-minor key ball in which the keys a tritone apart point in exactly opposite directions.

Consecutive bars produce a sequence of unit vectors that visualize the tonalities and modulations in 3D space: the stronger the modulation, the larger the angle between the vectors. The mapping of music begins always from the point of origin $(0,0,0)$ in three-dimensional space. This sequence of vectors is enriched by depicting each note in a bar with a smaller note vector of its own: the starting point for the note vector is determined by its temporal position in the bar and its length based on the note duration. The colors and directions of note vectors are defined according to the note’s pitch-class value. Pitch classes are here tied to the corresponding major key properties, that is to say, to their colors and directions. For example, if the note-onset time of a note—being a middle C (with a MIDI value of 60)—in the first bar is 2048 MIDI ticks and the time signature is $4/4$ (with the length of the whole bar being 4096 ticks), then the note vector starts from the midpoint of the first unit vector and moves in the direction of the C Major (60 modulo $12=0$ , which is associated with the pitch class for C). As stated previously, the note vector’s end point—assigned via a particular pitch class color (in this case green)—is determined by the relative duration of the note. The size of the point is—for its part—determined by the pitch of the note: the lower the note, the bigger the point size. Unit vectors together with note vectors constitute a tree-like wireframe model, one which is thereafter

Three-Dimensional Score: Seeing Music, Hearing Sculpture

a)

b) Figure 2: a) Vector sequence model of W.A. Mozart’s 4th movement in Symphony nr. 40 (of a “wireframe” and the entire virtual sculpture); b) A plastic 3D printed version of the same mapping.

extended by applying kernel density estimation (KDE). The local magnitude of the expansion depends on the number of notes in the area. The map “layers” show a dot distribution to represent the number of notes in that particular area. In Figure 2a, we have allowed a maximum of three layers, in Figure 3, seven.

We call the end product of the mapping process a three-dimensional score since it offers a kind of symbolic notation in 3D space. Such virtual sculptures are not only artistically fascinating; they also offer interesting music analytical insights.

In Figure 2a, we see such a DNA-like wireframe inside a cover produced by kernel density estimation. The composition behind the virtual sculpture in Figure 2, is the final movement in Symphony nr. 40 by W.A. Mozart.

Figure 3: VSM mapping of Olivier Messiaen’s Apparition of the Eternal Church.

The quite straightforward, tube-like form of the figure means that Mozart’s composition mainly moves around in its basic tonal regions. However, as Arnold Schonberg has noted, the compass of modulation in

527

Karttunen and Tenkanen

the final movement’s development section is among the most remote in classical symphonies [2, 149]. The development section in the middle of the movement modulates violently along the key circle of fifths (9 steps out of 12) from F minor towards C# minor and then returns to G minor. This is seen as a bulge in the middle of the vector sequence model (see Figures 2a and 2b).

As a reference point of, another example of VSM mapping is shown in Figure 3. Olivier Messiaen’s music is not considered to be traditionally tonal. Even though we cannot define actual major and minor keys based on Messiaen’s music, VSM mapping shows that there are tonal centers that remain the same over several bars.

Conclusions

The Vector Sequence Model provides artists with a strong analytical and artistic tool. For example, it allows us to improvise a three-dimensional virtual sculpture using a MIDI keybord. The end product—in which the cover forms the sculpture surface—can be converted into STL and PLY format for 3D-printing (Figure 2b). In the current model we have concentrated on tonalities of music. However, chord structures and rhythmic information could be studied as well.

References

[1] Krawczyk, Robert J., Exploring the Visualization of Music, in Proceedings of Bridges 2012: Mathematics, Music, Art, Architecture, Culture (2012), pp. 393–396.

[2] Schönberg, Arnold, Structural Functions of Harmony, Edit. by L. Stein, London (1969).

[3] Kernel density estimation, https://en.wikipedia.org/wiki/Kernel_density_estimation, read 13.3.2016.

[4] Krumhansl, Carol L., Cognitive Foundations of Musical Pitch. Oxford University Press. New York, (1990).

[5] Wickelmaier, Florian, An introduction to MDS, Sound Quality Research Unit, Aalborg University, Denmark (2003), https://homepage.uni-tuebingen.de/florian.wickelmaier/ pubs/ Wickelmaier2003SQRU.pdf, read 28.4.2016.

[6] Krumhansl, Carol L. & Kessler, Edward J. Tracing the Dynamic Changes in Perceived Tonal Organization in a Spatial Representation of Musical Keys, Psychological Review 89 (1982), pp. 334–368.

[7] Tenkanen, Atte, Comparison Structure Analysis, Doctoral thesis, Turun yliopisto, Annales Universitatis Turkuensis Ser B 327, Humaniora (2010).

3By chord structures, we mean here so-called “set classes,” which are defined in musical set theory and which can be segmented based on MIDI data as well (see, for instance, [7]).