Move it or Lose it: Derivision, the Discrete Time Derivative of a Video

Year: 2020 Authors: Markus Juvonen; Saara Lehto; Tuomo Rainio; Juulia Lahdenperä

Core claim

Frame differencing can be used as an accessible creative and educational method for revealing movement in video and linking mathematical change to artistic expression.

Topics

frame differencing, movement visualization, interactive workshop, creative coding

Domains

discrete derivatives, finite differences, computer vision, visual arts, dance, communication, digital video art

Methods

live camera feeds, mini-labs, absolute difference, blurring and thresholding

Media

video frames, Processing, camera+laptop setup, digitally processed photographs

Paper text

The text below is the locally extracted OCR/Markdown version of the paper. Raw PDF files remain local and are not published here.

Bridges 2020 Conference Proceedings

Move it or Lose it: Derivision, the Discrete Time Derivative of a Video

Markus Juvonen${}^1$, Saara Lehto${}^2$, Tuomo Rainio${}^3$ and Juulia Lahdenperä${}^4$

${}^1$Dept. of Math. and Stat., University of Helsinki, Finland; markus.juvonen@helsinki.fi ${}^2$Dept. of Math. and Stat., University of Helsinki, Finland; saara.lehto@helsinki.fi ${}^3$Academy of Fine Arts, Uniarts Helsinki, Finland; tuomo.rainio@uniarts.fi ${}^4$Dept. of Math. and Stat., University of Helsinki, Finland; juulia.lahdenpera@helsinki.fi

Abstract

In this interactive hands-on workshop, you get to experience an immersive combination of mathematics, visual arts, dance, and communication through Derivision. Derivision is a process of taking a discrete time derivative of a video which opens up a joint purpose for these seemingly separate fields. In short, Derivision captures your movement - but if you stop, you become invisible. The workshop consists of four mini-labs, after which you get to work in small groups on your own Derivisions.

Introduction

It all started with a very simple question: how would you dance a derivative? We were a group of people on our way to a science video minicourse pondering about the possible topics for our video. The group consisted of a communications expert accompanied by three mathematicians, each having additional expertise in photography, dance, and music. Instead of dancing a derivative, we ended up differentiating dance [1]. With differentiating we simply mean the discrete derivative - the difference - of two consecutive video frames. The creative process was inspired by Tuomo Rainio’s video artworks [3], and in this workshop, the original idea is developed further to create a real-time immersive experience on mathematics, visual arts, dance, and communication. Through our four mini-labs, A peek under the hood, Light makes difference, Dance is all about movement, and Conversation station, the participants get a hands-on experience on Derivision and are exposed to the simultaneous presence of arts and science. The inspiring lure of Derivision becomes evident during the workshop through the participants’ small projects utilising a customised code for calculating the frame difference from live camera feeds.

The Workshop

The workshop can cater a maximum of 20 participants. The 90 minutes allocated for the workshop are divided as follows: 10 minutes for orientation, 40 minutes for visiting mini-labs, 25 minutes for working on own Derivisions, and 15 minutes for video gala and reflection.

For a successful implementation of the workshop, the following is required from the conference organizers:

• a large workspace (for approximately 40 people) that allows participants to move around freely and to sit down by tables to work in 4 groups
• at least one video projector and film screen or a large TV screen with HDMI connectivity.

The workshop instructors will provide three camera + laptop combinations with all necessary software installed to be used during the workshop. One camera will be set up to point straight down at a table. The other two cameras will simply capture scenes from the side so built-in laptop webcams will work just fine. The participants do not need to bring anything with them, and previous knowledge on making videos or programming is not required.

Juvonen et al.

Orientation

The workshop starts with a short orientation to the mathematical and artistic ideas behind the workshop, including some of the topics discussed below. The rest serves as background material for the reader and will be learned through the workshop activities by the participants.

A digital video is composed of a sequence of still image frames. The frame rate of a video is usually expressed in frames per second (fps) and is commonly in the range of 24-30 fps. As we are dealing with 8 bit RGB-color images, the images will have three color channels. Each image pixel has one intensity value ranging from 0 to 255 for each channel. Therefore, the pixel size of an image is width $\times$ height $\times$ color channels and is most commonly expressed in matrix form. For the sake of simplicity, only one color channel is selected for the current purpose, resulting in a grayscale image of the size width $\times$ height to work on.

To compute the difference of consecutive frames, the corresponding pixel values can simply be subtracted. Figure 1 shows an example of calculating the difference of two $4\times 4$ grayscale images. As the primary interest is the magnitude, not the direction of the change, the absolute value of the difference, abs (frame2 - frame1) is computed. In this way, the resulting matrix has its pixel values in the same range saving the trouble of normalizing the result before viewing it. If the value of a pixel stays the same from one frame to the next its value is 0 which corresponds to black. The larger the change in the pixel value, the brighter it appears in the difference image. If the direction of the change is of interest, the negative values can for example be mapped to one color and the positive ones to another.

In the computer vision community, the basic algorithm behind the idea of Derivision is called frame differencing; it is an essential tool used for example in video compression applications where the redundancy in the information between video frames can be exploited to save the amount memory needed to encode each frame [6]. In motion detection applications convolution with a blurring kernel and a thresholding operation are often applied to distinguish the movement from noise in the captured video. Also, instead of tracking the difference of consecutive frames, the subtraction of the background of a scene is sometimes beneficial for detecting movement [7]. Figure 2 shows the result of the same computation as before, applied to a pair of $1024\times 1024$ pixel images including the effect of blurring and thresholding.

Figure 2: Two image frames, their absolute difference, and the result after blurring and thresholding.

Move it or Lose it: Derivision, the Discrete Time Derivative of a Video

Figure 3: Tuomo Rainio: Garden (2006), $38\times 50$ cm, digitally processed photograph printed on offset film.

Visual artist Tuomo Rainio has applied the idea of time derivatives to photography and video processing. Rainio’s work is based on the idea of representing time through changes over time. Instead of showing a photograph that freezes the moment, Rainio has composed the images from two photographs depicting two moments (Figure 3). As a result, human figures appear in the scene both absent and present.

Figure 4: Single frame from Tuomo Rainio: Pont Austerlitz II (2012), digital video HD, 4 min. 15 sec.

In Rainio’s video works, this idea is developed further; when derivative calculation is repeated over the whole duration of the video sequence, the traces of people moving accumulate in the city landscape [4] or the tracks of flying birds draw out the landscape depicting a bridge and riverbanks (Figure 4). These artworks were created using Processing [2], a creative coding framework that will also be used in the mini-labs of the workshop.

Visiting Mini-labs

After the orientation, the participants are divided into four groups in which they circulate in the mini-labs and explore different aspects of Derivision. All groups get to explore all of the following four mini-labs:

• A peek under the hood. We will discuss some technical details such as resolution, bit depth and frame rate and examine how to isolate the movement information from the video frames. The participants get to see the algorithms and software code behind the differentiated video feed and to appreciate how simple computations highlight the changes over time. The goal is to understand the central mathematical parts of the code and to explore how modifications to the parameters affect the resulting video. The participants are also encouraged to suggest possible adjustments to the code to create their own custom Derivision look. The original code is freely available on GitHub [5].
• Light makes a difference. This mini-lab will introduce how artistic practice can be enriched with computation and a simple derivative calculation. With a video camera filming a table from above the participants will get the chance to explore collage technique and painting with light. An image with detailed descriptions of the setup can be found in the README section of [5]. Participants are provided with printed image material, flashlights and a mindset to explore, combine and mix images just by pointing a flashlight on them. This creative method is made possible by background subtraction that calculates the derivative between empty background and the current scene. Hand gestures, light drawings and printed images merge together in a painterly manner.
• Dance is all about movement. In this mini-lab, we will study our own body movement – and stillness. The participants get to stand and move in front of a camera and view their differentiated image in real-time. The setting is ideal for getting a kinesthetic understanding of the Derivision. Here you get to feel how movement affects the derivative, how sensitive it is and how easy or not it is to be invisible. Other interesting points of study can be: Is it possible to move a part of your body while another part of you stays invisible? How do your surroundings and your clothes affect the image? Is it possible to appear to be standing still in the image and still be visible? What kinds of illusions can you create on your own or with a partner?
• Conversation station. What do arts and mathematics aim at communicating? At first, the question seems a little odd: mathematics and arts communicate to different audiences, they are motivated by different aims and they use different means of communication. In fact, arts and mathematics seem to offer distinctive lenses through which the world looks quite different. In this discussion-based mini-lab, the participants get to elaborate on their ideas on science and arts communication and bring these worlds together by addressing different forms of communication accessible through Derivision. What would the participant like to communicate with Derivision and to whom? In essence, this mini-lab acknowledges that art is to be experienced – but how to experience science?

Working on Your own Derivisions

What would you like to differentiate? At this point, the participants have explored different aspects of Derivisions and are ready to make their own small Derivision projects. In the project, it is possible to delve deeper into one of the mini-lab themes or combine several ideas to create a unique work. First, the groups will plan and practice their video with an approximate length of 30 to 60 seconds. Then, the videos are

recorded with an easy-to-use tool and stored for viewing and possible sharing. The workshop instructors are there to help with technical and creative aspects of the process.

Video Gala and Reflection

Finally, after the participants have finished recording their own Derivisions, all the videos are presented to the whole workshop in a short video gala. The workshop ends in a reflective discussion where the participants get to share their ideas and elaborate on their experiences. The participants also get to give feedback on the workshop for future improvements.

Reflections and Conclusions

Mathematics, visual art and dance all study movement and change. The interesting research questions of each field can seem separate and disjointed, but what happens if all three fields are working towards a joint purpose? During this project, we found a common language and vibrant currents of communication between all three fields. We believe that together these fields can create something new and unique; something that could not have been a product of any one of the fields alone and furthermore, something that could prove interesting and meaningful to each one of the fields as well.

In Derivision, mathematics forms the core of the concrete programming tools. Differentiation is the mathematical instrument for studying change and it is ubiquitous in the field and numerous applications. For a mathematician, Derivision offers a powerful tool for visualization. The ability to interactively inspect how an algorithm works opens up opportunities for deeper learning and understanding of the mathematical concepts involved and can eventually lead to new discoveries.

However, the idea of Derivision is not unique to mathematics; this project is greatly inspired by Tuomo Rainio’s visual art. Visual artists have the ability to find beauty and interest in the world that surrounds us and to interpret intellectual and even technical ideas into emotionally touching, approachable works of art. At the same time, the expansion of the creative coding movement to fine arts has brought coding accessible for artists, makers, and others with no professional training in computer science.

The combination of mathematics and visual arts is spiced up with dancers, the experts of changes in movement. Choreographers juxtapose movement and stillness for emphasis, and nuanced dance techniques enable various isolations that form different movement qualities and dance styles. In Derivision, both movement and stillness get new altered emphasis, which unfolds new tantalizing opportunities for dancers to explore: what happens to dance when we no longer see the stillness?

In Tuomo Rainio’s work Letter in Figure 5, two moments in time are present in the same picture. Together they reveal new shapes and worlds that would have remained invisible if not for each other. The same is true when mathematics, visual arts, dance and communication merge in a meaningful way – as they do in Derivision.

Acknowledgements

The authors would like to express their gratitude to Anni Aarinen for her valuable input as the communications specialist in the original science video team. The authors would also like to thank Professor Samuli Siltanen for providing the first version of the Derivision code. The authors are also grateful for all participants of the Tvärminne science video workshop for fruitful discussions and encouragement.

Juvonen et al.

Figure 5: Tuomo Rainio: Letter (2006), $38\times 50$ cm, digitally processed photograph printed on offset film.

References

[1] M. Juvonen, S. Lehto, J. Lahdenperä and A. Aarinen. “Dance derivative: differentiation brings out the movement”, YouTube, Helsingin yliopisto, 25.10.2019, https://youtu.be/mbnLepnSJ90. [2] Processing Foundation. Processing 3.5.4, as of Feb. 2020, available at http://www-processing.org. [3] T. Rainio. Artists selected works, http://www.tuomorainio.fi/works.. [4] T. Rainio. City, (2005), https://vimeo.com/77401271. [5] T. Rainio. TimeDerivative, (2020), GitHub repository, https://github.com/trainio/timeDerivative. [6] A. A. Rodriquez et al. “System and method for frame differencing video compression and decompression with frame rate scalability.” U.S. Patent No 5,384,598, 1995. [7] S. Shaikh, K. Saeed and N. Chaki. “Moving Object Detection Using Background Subtraction.” Moving Object Detection Using Background Subtraction. SpringerBriefs in Computer Science. Springer, 2014.