Unveiling the Invisible – Mathematical Approaches for Virtual Image Restoration in the Arts
Year: 2020 Authors: Carola-Bibiane Schönlieb
Core claim
Mathematics can reconstruct damaged or overpainted artworks from digital images in ways that are informed, objective, and useful for cultural heritage.
Topics
virtual image restoration, digital image processing, cultural heritage, image inpainting
Domains
partial differential equations, applied harmonic analysis, statistics, machine learning, art conservation, illuminated manuscripts, painting restoration, fresco restoration
Methods
image inpainting, patch-based methods, osmosis filtering, deep neural networks
Media
digital photographs, infrared imaging, illuminated manuscripts, painted frescoes
Paper text
The text below is the locally extracted OCR/Markdown version of the paper. Raw PDF files remain local and are not published here.
Bridges 2020 Conference Proceedings
Unveiling the Invisible – Mathematical Approaches for Virtual Image Restoration in the Arts
Carola-Bibiane Schönlieb¹
¹ Department of Applied Mathematics and Theoretical Physics, University of Cambridge; cbs31@cam.ac.uk
Abstract
The last fifty years have seen an impressive development of mathematical methods for the analysis and processing of digital images, mostly in the context of photography, biomedical imaging and various forms of engineering. The arts have been mostly overlooked in this process, apart from a few exceptional works in the last ten years. With the rapid emergence of digitisation in the arts, however, the arts domain is becoming increasingly receptive to digital image processing methods and the importance of paying attention to this therefore increases.
Virtual image restoration, also called image inpainting, denotes the process whereby missing or occluded parts in images are filled in based on some a-priori information that is, e.g., provided by the intact parts of the image. In this talk I will sketch and motivate different mathematical principles that can guide a digital restoration attempt. Digital photographs of art pieces are essentially mathematical objects, and this puts the vast toolbox of mathematics at the restorers’ fingertips.
We will encounter the role of differential equations, patch-based methods and deep learning for virtually restoring structure, texture and colour in images. In particular, we will show examples from the restoration of the Neidhart frescoes (Tuchlauben, Vienna), the restoration of a painting by Sebastiano Del Piombo (the Hamilton Kerr Institute, The Fitzwilliam Museum), and the unveiling of hidden structures in illuminated manuscripts revealed by infrared imaging (part of the MACH project¹). After a critical discussion of restoration results I will conclude by pointing out the capabilities and limitations of digital restoration methods, and provide some hints towards other applications of mathematics in cultural heritage, including paint layer and pottery classification.
While admiring a work of art – possibly damaged by wear-and-tear or altered by human intervention – we have all played at least once the part of an art restorer and interpreter. “What did the art piece look like when it was created?” and “What materials were used to make it and have they changed over time?” are only two of the questions we may have asked ourselves. Mathematics can play an important role for answering them in an informed while objective manner.
Indeed, there is a myriad of mathematical methods sourced from partial differential equations, applied harmonic analysis, statistics and machine learning designed for virtual restoration of digital images, some of which made their way to the arts and cultural heritage conservation². Figure 1 shows an example from [4] on the digital restoration of Claude de France’s Primer (c. 1505, MS 159³). Here, the picture on the left shows an illumination from the Primer which illustrates the story of Adam and Eve in the garden of Eden. The two figures were originally depicted naked, as described in the book of Genesis but a later owner wanted them clothed with additional veils, leaves or beast skin added in the illumination. The use of infrared imaging as shown in the middle left picture in Figure 1 allows to look through these added layers, unveiling hidden structural information underneath the painted layer. Marking the area in the image we want to “unveil” and highlighting the structures we can see in the infrared image in red in the middle right picture in Figure 1, we can use so called osmosis filtering [10, 7] and patch-based inpainting [1] to fuse the details appearing in the
¹http://www.mach.maths.cam.ac.uk ²http://blogs.springeropen.com/springeropen/2018/11/12/unveiling-the-invisible-mathematical-methods-for-restoring-and-interpreting-illuminated-manuscripts/ ³For more information see http://www.fitzmuseum.cam.ac.uk/illuminated/manuscript/discover
Schonlieb
Figure 1: Example of a digital restoration result for illuminated manuscripts from [4]. Fitzwilliam Museum, MS 159 Folio 4r, ©Fitzwilliam Museum, Cambridge.
near infrared reflectogram with the colours of the visible colour image, in particular the skin tones, to create a digital version of the illuminations as they could have looked before overpainting, cf. the rightmost image in Figure 1.
In my talk I will be discussing some of the main mathematical mechanisms that make digital restorations as in Figure 1 possible. The discussion will range from partial differential equations [8, 2, 3], to applied harmonic analysis [6, 5], to texture synthesis and so-called patch-based methods [1], and deep neural networks [9].
Acknowledgements
The author acknowledges support from the Leverhulme Trust project Unveiling the Invisible, the EPSRC grants EP/M00483X/1, EP/N014588/1, EP/K009745/1, the Alan Turing Institute TU/B/000071, CHiPS and NoMADS (Horizon 2020 RISE project grant), the Isaac Newton Institute, and the Cantab Capital Institute for the Mathematics of Information.
References
[1] P. Arias, G. Facciolo, V. Caselles, G. Sapiro, “A variational framework for exemplar-based image inpainting”, International Journal of Computer Vision, 93(3), pp. 319–347, 2011. [2] W. Baatz, M. Fornasier, P. Markowich, C.-B. Schonlieb, “Inpainting of ancient Austrian frescoes”, Conference Proceedings of Bridges, pp. 150-156, 2008. [3] W. Baatz, M. Fornasier, P. Markowich, C.-B. Schonlieb, “Binary Based Fresco Restoration”, Conference Proceedings of Bridges, 2009. [4] L. Calatroni, M. d’Autume, R. Hocking, S. Panayotova, S. Parisotto, P. Ricciardi, C.-B. Schonlieb, “Unveiling the invisible: mathematical methods for restoring and interpreting illuminated manuscripts”, Heritage Science, 6 (56), 2018. [5] B. Cornelis, T. Ruzic, E. Gezels, A. Dooms, A. Pizurica, L. Platisa, J. Cornelis, M. Martens, M. De Mey, I. Daubechies, “Crack detection and inpainting for virtual restoration of paintings: The case of the Ghent Altarpiece”, Signal Processing 93, no. 3, pp. 605-619, 2013. [6] I. Daubechies, Ten lectures on wavelets, Vol. 61, SIAM, 1992. [7] S. Parisotto, L. Calatroni, A. Bugeau, N. Papadakis, C.-B. Schonlieb, “Variational Osmosis for Non-linear Image Fusion”, arXiv:1910.02012. [8] C.-B. Schonlieb, Partial differential equation methods for image inpainting, Vol. 29, Cambridge University Press, 2015. [9] D. Ulyanov, A. Vedaldi, V. Lempitsky, “Deep image prior”, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9446–9454, 2018. [10] J. Weickert, K. Hagenburg, M. Breuß, O. Vogel, “Linear osmosis models for visual computing”, in International Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition, Springer, Berlin, Heidelberg, pp. 26-39, 2013, August.