Sonifying Games

Year: 2022 Authors: Donald Spector

Core claim

Competitive games can be sonified effectively by mapping discrete moves to sounds that emphasize game motion rather than proximity to an endpoint.

Topics

sonification, competitive games, musical mapping, game states, auditory display

Domains

combinatorics, state spaces, lattice models, music, sound art, auditory visualization

Methods

move-to-sound mapping, pitch mapping, rhythmic mapping, melodic sonification, percussive sonification

Media

draughts match records, contract bridge bidding and play, Mathematica-generated audio, MP3 files, PDF scores

Paper text

The text below is the locally extracted OCR/Markdown version of the paper. Raw PDF files remain local and are not published here.

References

Sonifying Games

Donald Spector

Abstract

Using examples come from draughts and contract bridge, I explore the sonification of competitive games. We will see how different aspects of game play can be highlighted by different choices and, in so doing, how to create compelling sonic experiences, both in their own right, and as a basis for elaborated musical compositions.

Introduction

When, at a previous Bridges Conference, I presented a paper exploring the use of sonification to represent the process of and intermediate states encountered in solving puzzles [1], the question was raised as to what kind of results would ensue if similar ideas were applied to games—not the same specific sonification rules (given systems with different state spaces, this is not even a sensible notion), but the approach of representing via sounds the process of and intermediate states encountered in playing a game. This paper is is the result of exploring that question.

In sonification, the representation of data via sounds [2], one might be more interested in sonification as a tool to convey information about that data or more interested in sonification as a means to generate aesthetically meaningful experiences. My focus here is on the latter, though doing so with methods that keep some characteristics of the game play comfortably recognizable auditorily.

The data environment of games is quite different from that of mathematical puzzles. One may draw the distinction roughly as follows: a puzzle has a single agent moving through the state space of the puzzle, based on uncontested decisions, and the form of the final state (the “solution”) can be anticipated. In competitive games (our focus here), there are multiple agents, each independently causing movement through some state space, with competing goals, and while the rules specify when a match is over, that is not the same as having a pre-determined solution toward which one is pushing. For example, checkmate is a condition satisfied by a huge number of states, whereas the final state of a Rubik’s cube is fully determined in advance.

As discussed below, this difference in structure also shows up in the formulation of effective sonifications of competitive games. Rather than attempting to seek a sonification that tracks closeness to a conclusion, the most compelling sonifications of games, in my experience and as I discuss below, are ones that convey something of the kinetic nature of a game, the movement as it is played, rather than nearness to its conclusion.

For the purposes of this paper, I will focus on two games: draughts (also known as checkers) and contract bridge [3]. This will give us a chance to explore sonifications of differing game play spaces, and to reflect on the nature of the outcomes of different sonifications in these arenas. Given the kinds of sonifications I am interested in exploring, in which particular moves are mapped into particular sounds, games with discrete moves are needed, as draughts and contract bridge both are, and not games in which the play requires an analogue description, as is the case for most physical sports. One of the additional reasons for using these two games is that their game play is sufficiently limited to make sonification worth pursuing; too many constraints, of course, and the resulting sounds are likely to be dull, but if game play is too varied, the possibilities of what to sonify and how are so large that the result could be, crudely speaking, almost anything. The old adage that constraints lead to creativity remains true in this context.

This paper will introduce a number of systems of sonification, and reflect on the kinds of sonic experiences they provide. Accompanying this paper are files with examples of what these methods can achieve. Given the many ways these sonification systems can be applied and combined, neither this paper nor the examples are intended to be exhaustive, but they should situate the reader comfortably in the field of what is possible.

As a matter of terminology, going forward, I will use “game” to refer to the overall entity specified by the rules, and the term “match” to refer to an individual instance of a game. Also note that the implementations of the sonifications in the accompanying files were done in Mathematica, and so references to melodic or percussive musical instruments refer to the software rendering of those instruments in Mathematica.

Draughts a/k/a Checkers

Draughts, also known as checkers, is a game between two players, known as black and white [3]. Without giving all the rules, I highlight here the essentials needed for our purposes. The game is played on the black squares of an $N\times N$ board with alternating black and white squares; the most common variants are $N=8$ and $N=10$. Black and white start with their pieces on opposite sides of the board. In most turns, each player advances a piece diagonally forward one square, though captures are executed by leaping over an opponent’s piece, a single piece can make multiple consecutive captures in a single turn, and once a piece reaches the opponent’s first rank, it becomes a king and can subsequently move in either direction. Our goal is to explore ways to associate a sound with each move, and then the sonification of a given match will just be the sequence of sounds associated with the sequence of the players’ moves. We will do this in two fundamentally different ways. Taking a page out of the study of spin systems, which can be described by the states of spins on individual lattice sites or by how the spin changes as one traverses a link or bond between adjacent spins [4], we will look at sonifications based on the locations of pieces and sonifications based on the vectors that describe the displacement of a piece when it moves. The accompanying files provide a handful of examples of what I discuss below.

For an $N\times N$ draughts board, we will use games recorded using Portable Draughts Notation [5] in which the black squares are numbered from 1 to $N^2/2$, and each move is indicated by an ordered pair of numbers, the first of which labels the starting square of a piece, and the second of which labels the square it ends on. Ordinary moves and captures are notationally distinguished, but we will ignore this for simplicity.

The first sonification we will explore assigns each square a note, and thus each move a sequence of two notes. In order that the notes correspond in a meaningful way to the position of squares on the board, and given the widespread familiarity with the scales of Western music, we will pick a base note for square 1, and then assign to the square $n$ the note $n-1$ half-steps up from the base note in the standard, well-tempered chromatic scale. (One could, of course, use other scales.) Each move, then, is associated with a pair of sounds. Note that the overall pitch indicates which side of the board the move is taking place in, while whether the two pitches in a move are ascending or descending indicates which direction a piece is moving. One can make various choices such as the instrument playing the notes and the duration of the notes. The resulting melodic representations of the draughts matches have the familiar sound of contemporary atonal pieces of music. In trying different schemes, I found that giving the initial square note a shorter duration than the final square note (say, 0.1 and 0.3 seconds, respectively) created an effective musical experience that most captured something of the feel of game play. A major reason for this is that short/long duration pairing makes it easy, even in the middle of a given sonification, to hear which pairs of notes constitute a single move. One can add additional features on top of this, such as using different instruments for the two players, or different instruments for the starting and final notes of a move (this latter can, with suitable choices, create the sense of two interleaved pieces of music). Using different instruments to represent each player’s moves creates a sense of the dynamic between the two players, while musically, we get a piece for two instruments interacting and responding to each other.

Sonifying Games

The second, and to my mind and taste, perhaps more interesting, sonification is one that hinges on the geometric form of moves in draughts. Each move has a starting and ending point, and so each move has associated with it a displacement vector; we call this the move vector, and it is an ordered pair $(\Delta c,\Delta r)$ where $\Delta c$ is the change in the column number, and $\Delta r$ the change in rank, with direction in each case associated with the sign of the number. Note that for ordinary moves in draughts, the move vector will be one of the four vectors $(\pm 1,\pm 1)$, though with captures one can find other move vectors, such as $(2,-2)$ or $(0,4)$. By focusing on move vectors, we are focusing on the structure of the game play.

One resulting challenge is that the majority of move vectors in a game correspond one of just four possibilities. This challenge turns out to be an opportunity: whereas this limited set might make a melodic sonification dull, it works very well for a percussion-based sonification. In particular, associating each move vector with a distinct percussive sound (e.g., the striking of a hi-hat or of a snare drum) allows us to convert a game of draughts—that is, its sequence of moves—into a sort of drum/percussion solo. The resulting sonifications are not only appealing, but also accessible to audiences unfamiliar with atonality. They create a sonic experience with a regular rhythm and, to my ears, real driving energy. In the accompanying files, one can hear not only some examples of these percussion-based sonifications, but also hear that the combination of the two sides’ moves creates a more interesting musical entity than sonifying either side’s moves alone.

Contract Bridge

Contract bridge is a card game among four players with a standard deck of 52 cards, with 13 in each suit. The highlights of the rules [3] for our purposes are as follows. Play begins with a bidding phase, which I will return to at the end of this section. The next phase is the trick phase, with thirteen turns; in each turn, one of each player’s cards is played, from the same suit if possible. At the end of the game, all 52 cards from the deck have been played.

To sonify the game play, I used records of matches represented in Portable Bridge Notation [6]. For the trick phase, this notation is just a series of 52 cards being played, each card represented by one of the 13 characters $\{A,2,3,4,5,6,7,8,9,T,J,Q,K\}$ (for the card values ace through king, respectively) and one of the suits $\{C,D,H,S\}$ (for the suits $♣$, $\diamond$, $♡$, and $♠$, respectively). The 13 card values can be replaced by the numbers $\{1,2,3,4,5,6,7,8,9,10,11,12,13\}$, which allows for a very natural identification between card numbers and the notes of a chromatic scale, for example, associating 1 with $C$, 2 with $C$#, and so forth, up till 13, for the king, an octave above 1. Since the cards have a rank ordering in bridge, assigning the numerical values in a way that this numerical ordering is also a pitch ordering helps capture the meaning of the game play sonically. Setting aces to 1, although they outrank kings, is useful, as in this way, it is the two highest ranking cards—kings and aces—that are assigned to the same note.

The suits should in some way modify these notes. In our first sonification choice, each suit is associated with a different instrument (thus a $5♠$ might be an $E$ played on a violin, while a $5♠$ might be an $E$ played on a flute). Since each trick usually involves cards of (predominantly) one suit, this will give most tricks a coherent instrumental timbre, and when a player cannot follow suit, that shows up in a timbre mismatch. This method creates melodies that feel natural in the context of modern music, and have the structure of generalized serial tone row; we have four intermeshed tone rows (each note of the chromatic scales appears four times), each voiced by a different instrument, with the added twist that these are 13-note tone rows, so the fundamental pitch appears eight times, since the pitches for aces and kings are an octave apart.

In this type of sonification, I view the link from a given bridge match to a series of notes as a way to generate one of these generalized tone rows; as with Schönberg’s twelve-tone method [7], once the contract bridge tone row is generated, it is up to the composer to manipulate it for musical effect, making choic

1One can use a non-standard scale with 13 different notes, e.g., a well-tempered scale with notes differing by pitch ratios of 2^{1/13} . I have done this (see the accompanying files for an example), but found this more interesting intellectually than musically.

about duration of notes and what octave to place them in, and manipulating the tone row as it repeats using the inversion, retrograde, or transposition transformations, or combinations thereof. Thus, if one chooses to take this attitude, this type of sonification yields a foundation, not a final product.

An alternative is to use the card values for pitches as above, but to let the suit modify the duration of the note. Because players must follow suits when they can, this leads to clusters of like duration notes (say, ♣ cards as eighth-notes and ♢ cards as quarter notes, etc.), and so it creates atonal melodies with rhythmic coherence. (Percussion-based sonifications, as implemented with draughts, did not appear to produce particularly memorable results.)

Finally, let me add that I explored sonifying the bidding phase (see [3] for a description of the details of this phase). As it turned out, the bidding phases did not have enough structure to produce interesting sonifications. My most extensive attempt was to try to associate bids with note combinations in a way that the increasing bids corresponded to harmonic changes. The form of the bidding phase (typically, there are just a few bids, one pair of players does not bid much, and any possible bid can be the endpoint), however, just did not have a structure to yield results that were musically interesting. Even with concatenating several bidding phases, my attempts to create interesting sonifications were unsuccessful.

Conclusions

In this paper and the accompanying musical files [8], I have presented some approaches to the sonification of games. The particular examples have been taken from draughts and contract bridge. We have seen ways to make melodic and percussive sonifications, producing results that in some cases can be treated as ends in their own right, and sometimes as musical objects to be manipulated further. It is also the case that not every game produced equally interesting sonifications. While my methods required that the game play have discrete moves, having discrete choices is not sufficient. Chess is an example of a game in which there are so many different kinds of moves that broadly capturing game play simply has too many variables to produce auditorily coherent outcomes. (One can, of course, ignore certain variables to avoid some of this issue.) The scope of draughts and bridge seem to land at a sweet spot of being able to capture aspects of the structure of the game without being too constrained or too free, and thus able to yield musically worthwhile sonifications.

References

• [1] D. Spector, “The Tower of Har(mo)noi,” in D. Swart, F. Ferris, E. Torrence, ed., Proceedings of Bridges 2021: Mathematics, Art, Music, Architecture, Culture, Tessellations Publishing, 2021, pp. 257-260.
• [2] G. Kramer, ed. Auditory display : sonification, audification, and auditory interfaces. Proceedings of the Santa Fe Institute, vol. XVIII. Addison-Wesley, 1994.
• [3] Richard L. Frey. According to Hoyle. Ballantine Books, 1996.
• [4] Kramers, H. A., and G. H. Wannier. 1941. “Statistics of the two-dimensional ferromagnet.” Physical Review 60: 252-262.
• [5] https://en.wikipedia.org/wiki/Portable_Draughts_Notation. Accessed 28 Feb. 2022.
• [6] See materials at https://tistis.nl/pbn/. Accessed 28 Feb. 2022.
• [7] D.M. Randel, ed. The Harvard Dictionary of Music, 4th edition. The Belknap Press of Harvard University Press, 2003.
• [8] These mp3 files along with records of match moves and PDF scores can be found at https://people.hws.edu/spector/SonifyingGames/SonifyingGamesFiles.zip.