Exploring the Visualization of Music
Year: 2012 Authors: Robert J. Krawczyk
Core claim
Music can be visualized as compact geometric and architectural forms that preserve its structure while revealing aesthetic possibilities.
Topics
music visualization, MIDI data, fabricated forms, architectural abstraction
Domains
geometric representation, spatial coordinates, data mapping, sculpture, architecture, visual art, digital fabrication
Methods
piano roll diagrams, 2D and 3D mapping, shape variation, circular frameworks
Media
MIDI files, prints, laser-cut layers, 3D prints
Paper text
The text below is the locally extracted OCR/Markdown version of the paper. Raw PDF files remain local and are not published here.
Bridges 2012: Mathematics, Music, Art, Architecture, Culture
Exploring the Visualization of Music
Robert J. Krawczyk
College of Architecture
Illinois Institute of Technology
S. R. Crown Hall, 3360 South State Street
Chicago, Il, 60616, USA
E-mail: krawczyk@iit.edu
Abstract
Data can be used to generate artistic, sculptural and architectural forms. Each data type such as, numeric, text, sound, temperature, or wind is unique and can suggest a form or a method to visualize which is beyond the actual meaning and purpose of the data itself. The data can be viewed two dimensionally or three dimensionally. At times the relationship of data to physical form is obvious; at other times it is very subtle and abstract. The challenge is to develop both at the same time and to attempt to extract the beauty in the data itself. Music is another such set of data.
Overview
A number of my print, sculptural, and architectural pieces have dealt with the interpretation or the physical manifestation of data ranging from the scientific to the everyday. Figure 1 demonstrates the use of weather data and Figure 2 uses the color values within an image.
Figure 1: Chicago January temperature as a perforated panel and three-dimensional forms.
In the case of weather data, hourly temperature data for a single month is represented as perforations on a panel. The radius of each perforation is proportional to the temperature value. The value of the temperature can also be used as a radius and translated with cylindrical or spherical coordinates to create three-dimensional forms. In the second example, the image of clouds is converted from true color to grayscale; then the grayscale value of each pixel in the image is used as the height on a surface mesh.
Krawczyk
Figure 2: An image of clouds as three-dimensional surface.
Music
Music is another example of such set of data. A classic example is the Walt Disney’s 1940 animation Fantasia. Its first musical piece, Johann Sebastian Bach’s “Toccata and Fugue in D Minor”, includes animation of abstract patterns of color based on sound frequency values. Animation has a continuous property of time as does music. This research explores how such data can be made static and in a form that could be fabricated as sculptural or architectural elements.
Music can be expressed digitally in a MIDI file format. A MIDI file contains for each individual instrument each note played including its frequency, timing, and velocity. The MIDI format describes the instructions for the playing of a tune; not the actual sound made, as in an MP3 file. Preliminary explorations of this data have shown that there is a large rich set of physical forms possible using such simple data. Color and frequency is just one approach; other forms can be also investigated: starting with simple bars as in a piano roll, note frequency as elliptical and rectangular forms, and elliptical rings in a circular arrangement. Both two dimensional and three dimensional representations have been explored.
The Chan [1] report includes an excellent overview of different ways music has be visualized. It includes two-dimensional image examples and also real-time animation approaches. Most are fundamental in their visualizations or the artistic aspects are not extensively investigated. Many of the methods outlined try to determine the actual musical structure of the score itself. Wattenberg’s [2,3] two-dimensional Arc Diagrams are an excellent example of exploring structure in an artistic fashion. More recent examples of visualization and structure include Clos [4] and Tymoczko [5].
From another perspective, Hansen [6] is one of the only examples I have found that has attempted to use MIDI values to suggest three-dimensional sculptural forms. Some use individual note values; others use a three-dimensional sound wave structural approach. Hansen’s work dates back to 1999.
The initial inspiration for this inquiry came from Ferschin [7], in 2001. From a MIDI file, the time, pitch, volume, and instrument were extracted and related to architectural dimensions, such as, position, length, height, and color. The resulting three-dimensional geometrical constructions were not very compelling but did spark an interest in music as form that continues today.
Sketches
For these set of sketches, we start with the children’s tune “Twinkle, Twinkle Little Star”. This popular tune is based on the French melody “Ah! vous dirai-je, Maman” (Ah! I shall tell you, mum), published in 1761. Mozart also wrote variations of it, (K. 265 / K. 300e), in 1778. Its MIDI file includes two tracks,
Exploring the Visualization of Music
twelve measures, four beats per measure, and total of 149 notes. Figure 3 displays the entire score in a piano roll graphic format; notes are represented as simple rectangular bars. This physical format was the initial representation.
Figure 3: Music score as a piano roll; notes as bars.
To be able to represent the score in some static fabricated manner, the length to the score needs to be more compact, otherwise the scale of notes becomes very small. Figure 4 shows the score at four measures in sheet music order, framed, and then converted to three-dimensions by assigning a height to each note. Each note is still a simple rectangular bar.
Figure 4: Music score in sheet music order, rectangular framework; notes as bars
Once a compact framework is established, a number of variations can be explored in how each individual note could be expressed geometrically. Figure 5 shows the notes as rectangles, as filleted rectangles, and as ellipses. The width and height of the note is determined by note duration and frequency. In all cases, the bars or rings mirror the actual notes being played; their relative location and sound; a direct connection to its source.
Figure 5: Shape variation of notes.
Krawczyk
The rectangular framework mimics traditional sheet music. Music has not only been recorded on sheet music, but on cylinders, piano rolls, discs, records, and CDs. Another natural framework that is also compact would be a circular one; a radial version of traditional sheet music. Figure 6 offers some variations on the circular framework using bars and ellipses. Another framework could be a spiral.
Figure 6. Circular framework with notes as bars and ellipses.
From these sketches color can be assigned and two-dimensional prints can be generated. The next step is to convert the lines into bars, rings, and frames so these pieces could be laser cut and layered or 3D printed.
This initial work covers a number of different forms and interpretative explorations on a simple children’s song. Other musical types could be explored: jazz, rock, country and western, to classical. Each may suggest a unique interpretation based on its inherent structure and complexity. The underlying question explored is “What can music look like?”
Visualizing music explores another dimension to an already rich art form.
References
[1] Winnie Wing-Yi Chan, A Report on Musical Structure Visualization, Leonardo, Department of Computer Science and Engineering Hong Kong University of Science and Technology Clear Water Bay, Kowloon, Hong Kong, August 2007. [2] Martin Wattenberg, The Shape of Song, www.turbulence.org/works/Song, 2001 [3] Martin Wattenberg, Arc Diagrams: Visualizing Structure in Strings, Information Visualization, 2002, pp. 110-116, 2002. [4] Laia Clos, Graphic Notation System, tomedictions.bigcartel.com, 2012. [5] Dmitri Tymoczko, Geometry of Music, dmitri.tymoczko.com, 2012. [6] Jan Henrik Hansen, Form Follows Funk, www.formfollowsfunk.com, 2006. [7] Peter Freschin, Translating Music into Architecture, Mathematics & Design Conference Proceedings. 2001