Inpainting of Ancient Austrian Frescoes

Year: 2008 Authors: Wolfgang Baatz; Massimo Fornasier; Peter A. Markowich; Carola-Bibiane Schönlieb

Core claim

Higher-order variational and PDE inpainting can reconstruct missing structures in damaged frescoes, with the Neidhart frescoes as a motivating case.

Topics

digital inpainting, fresco restoration, higher-order PDEs, variational methods

Domains

partial differential equations, calculus of variations, image interpolation, total variation, conservation and restoration, mural painting, cultural heritage imaging, visual arts

Methods

Cahn-Hilliard equation, higher order total variation, variational minimization, numerical reconstruction

Media

ancient frescoes, digital images, grayvalue images, wall paintings

Paper text

The text below is the locally extracted OCR/Markdown version of the paper. Raw PDF files remain local and are not published here.

163

Inpainting of Ancient Austrian Frescoes

Wolfgang Baatz * Massimo Fornasier† Peter A. Markowich‡ Carola-Bibiane Schönlieb§

Abstract

Digital inpainting methods provide an important tool in the restoration of images in a wide range of applications. We present mathematical methods with certain higher order partial differential equations for the inpainting of ancient frescoes. In particular we discuss the Cahn-Hilliard equation for the inpainting of binary structure and a higher order total variation approach. As an example for the preformance of our algorithms we consider the recently discovered Neidhart frescoes in Vienna.

Keywords: Image interpolation, Cahn-Hilliard equation, total variation minimization, fresco restoration 2000 Mathematics Subject Classification. 65K10, 65N21, 49M30, 68U10

1 Introduction

An important task in image processing is the process of filling in missing parts of damaged images based on the information gleaned from the surrounding areas. It is essentially a type of interpolation and is called inpainting.

In the course of an ongoing interdisciplinary project¹ the authors aim to use digital inpainting algorithms for the restoration of frescoes. Particular consideration have the new found Neidhart frescoes (Tuchlauben 19, 1010 Vienna). These ancient frescoes from the 14th century are depicting a cycle of songs of the 13th century minnesinger Neidhart von Reuental. Hidden behind a wall over years the frescoes got damaged in the process of exposure. Advanced mathematical tools were developed specifically for so-called “mathematical inpainting/retouching” of digital images. To this end, variational methods and third and fourth order partial differential equations have been investigated. Efficient numerical methods for the solution of the devised partial differential equations have been designed.

In the following we discuss our mathematical inpainting methods and present numerical results from their application to the Neidhart frescoes.

2 Neidhart frescoes

Fragments of 14th century wall frescoes found beneath the crumbling plaster of an old apartment in the heart of Vienna, depict a popular medieval cycle of songs of the 13th century minnesinger,

*Academy of Fine Arts Vienna, Institute for Conservation and Restauration, Schillerplatz 3, A-1010 Vienna. Email: w.Baatz@akbild.ac.at †Johann Radon Institute for Computational and Applied Mathematics (RICAM), Austrian Academy of Sciences, Altenbergerstrasse 69, A-4040, Linz, Austria Email: massimo.fornasier@oeaw.ac.at ‡Professor of Applied Mathematics, DAMTP, Centre for Mathematical Sciences, Wilberforce Road, Cambridge CB3 0WA, United Kingdom, and Adjunct Professor of Applied Analysis, Faculty of Mathematics, University of Vienna. Email: P.A.Markowich@damtp.cam.ac.uk §Department of Applied Mathematics and Theoretical Physics (DAMTP), Centre for Mathematical Sciences, Wilberforce Road, Cambridge CB3 0WA, United Kingdom. Email: c.b.s.schonlieb@damtp.cam.ac.uk ¹WWTF Five senses-Call 2006, Mathematical Methods for Image Analysis and Processing in the Visual Arts

Figure 1: Part of the Neidhart frescoes

Neidhart von Reuental. In the very late 14th century, Michel Menschein, a wealthy Viennese council member and cloth merchant, commissioned local artists to paint the stories in Neidhart’s songs on the walls of his Festsaal (banquet hall). The Neidhart frescoes provide a unique peek into medieval humor, and at the same time, a peek into the taste of a medieval man.

In Figure 1 a part of the Neidhart frescoes is shown. The white holes in the fresco are due to the wall which covered the fresco until a few years ago. They arised when the wall was removed. In the following we want to apply digital restoration methods to these frescoes. Thereby the main challenge is to capture the structures in the preserved parts of the fresco and transport them into the damaged parts continuously. Due to their great age and almost 600 years of living by owners and tenants in the apartment, saturation, hue and contrast quality of the colors in the frescoes suffered. Digital grayvalue, i.e., color interpolation, in the damaged parts of the fresco therefore demands sophisticated algorithms taking these lacks into account.

3 Methods

In the following we present the mathematical methods we used in order to reconstruct the damaged parts in the fresco. We begin with stating the mathematical variational approach for this task and roughly summarize existing inpainting methods of this type. In particular two inpainting methods based on higher order partial differential equations, i.e., Cahn-Hilliard- and $TV-H^{-1}$ - inpainting, are to be presented. We finalize this section by proposing a possible strategy to adapt these two inpainting approaches to the requirements of the Neidhart frescoes.

Given an incomplete image $f$ represented by an intensity function (or by a vector of intensity functions) in a suitable Banach space, defined on $\Omega \subset {\mathbb{R}}^2$ an open and bounded domain, the problem is to reconstruct the original image $u$ in the damaged domain $D\subset \Omega$ , called inpainting domain. In the following we are especially interested in so called non-texture inpainting, i.e., the inpainting of structure, like edges and uniform coloured areas in the image, rather than texture. In the pioneering works of Caselles et al. [6] (who called it disocclusion instead of inpainting) and Bertalmio et al. [2] partial differential equations have been first proposed for digital non-texture inpainting. The inpainting algorithm in [2] extends the graylevels at the boundary of the damaged domain continuously in the direction of the isophote curves to the interior via anisotropic diffusion. The resulting scheme is a

discrete model based on nonlinear partial differential equations:

$u_t={\nabla }^{\perp }u\cdot \nabla \Delta u,$

solved inside the inpainting domain $D$ using image information from a small stripe around $D$. The operator ${\nabla }^{\perp }$ denotes the perpendicular gradient $(-{\partial }_y,{\partial }_x)$. In subsequent works variational models, originally derived for the tasks of image denoising, deblurring and segmentation, have been adopted to inpainting. In contrast to former approaches, like [2], the proposed variational algorithms are applied to the image on the whole domain $\Omega$. This procedure has the advantage that inpainting can be carried out for several damaged domains in the image simultaneously and that possible noise outside of the inpainting domain is removed at the same time. The general form of such a variational inpainting approach is

$\hat{u}(x)={\mathrm{argmin}}_{u\in H_1(\Omega )}(J(u)={\int }_{\Omega }R(u)dx+\frac{1}{2}\Vert \lambda (f(x)-u(x)){\Vert }_{H_2(\Omega )}^2),$

where $f\in H_2(\Omega )$ (or $f\in H_1(\Omega )$ depending on the approach) is the given damaged image and $\hat{u}\in H_1(\Omega )$ is the restored image. $H_1(\Omega ),H_2(\Omega )$ are Banach spaces of functios defined on $\Omega$ and $R(u)$ is the so called regularizing term $R:H_1(\Omega )\to \mathbb{R}$. The function $\lambda$ is the characteristic function of $\Omega \setminus D$ multiplied by a large constant, i.e., $\lambda (x)={\lambda }_0>>1$ in $\Omega \setminus D$ and $0$ in $D$. Depending on the choice of the regularizing term $R(u)$ and the Hilbert spaces $H_1$, $H_2$ various approaches have been developed. The simplest model is the total variation model, where $R(u)=|Du|$ the total variation of $u$, $H_1=BV(\Omega )$ the space of functions of bounded variation and $H_2=L^2(\Omega )$, cf. [10, 8, 19, 20]. A variational model with a regularizing term of higher order derivatives, i.e., $R(u)=(1+\nabla \cdot (\frac{\nabla u}{|\nabla u|}))|\nabla u|$, is the Euler’s elastica model [7, 17]. Other examples are the active contour model based on Mumford and Shah’s segmentation [21], and the inpainting scheme based on the Mumford-Shah-Euler image model [11].

Now second order variational inpainting methods (where the order of the method is determined by the derivatives of highest order in the corresponding Euler-Lagrange equation), like total variation inpainting (cf. [20], [8], [9]) have drawbacks as in the connection of edges over large distances or the smooth propagation of level lines (sets of image points with constant grayvalue) into the damaged domain. In an attempt to solve both the connectivity principle and the so called staircasing effect resulting from second order image diffusions, a number of third and fourth order diffusions have been suggested for image inpainting.

A variational third order approach to image inpainting is the CDD (Curvature Driven Diffusion) method [9]. Solving the problem of connecting isophotes over a wide range the isophotes are still interpolated linearly. This has driven Chan, Kang and Shen to a reinvestigation of the earlier proposal of Masnou and Morel [17] on image interpolation based on Eulers elastica energy. In their work [7] (already mentioned earlier in this section) the authors present the fourth order elastica inpainting PDE which combines CDD and the transport process of Bertalmio et. al [2]. The isophotes are connected by minimizing the integral over the length and the squared curvature within the inpainting domain. This leads to a continuous connection of isophotes also over large distances. Of course this can also be interpreted via the second boundary condition, necessary for an equation of fourth order. Not only the grayvalues of the image are specified on the boundary of the inpainting domain but also the gradient of the image function, namely the direction of the isophotes are given. Further also combinations of second and higher order methods exist, e.g. [16]. The combined technique is able to preserve edges due to the second order part and at the same time avoid the staircase effect in smooth regions. A weight function is used for this combination.

The main challenge in inpainting with higher order flows is to find simple but effective models and to propose stable and fast discrete schemes to solve them numerically. A new approach in the class of fourth order inpainting algorithms is inpainting of binary images using a modified Cahn-Hilliard equation [3], [4]. In these works Bertozzi, Esedoglu and Gillette proposed an inpainting approach for binary images using a modified Cahn-Hilliard equation. The inpainted version $u(x)$ of $f(x)\in L^2(\Omega )$ is constructed by following the evolution equation

[ \left{\begin{array}[]{ll}u_{t}=\Delta(-\epsilon\Delta u+\frac{1}{\epsilon}F^{\prime}(u))+\lambda(f-u)&\text{in }\Omega,\ \frac{\partial u}{\partial v}=\frac{\partial\Delta u}{\partial v}=0&\text{on }\partial\Omega,\end{array}\right. ] (1)

with $F(u)$ is a so called double-well potential, e.g., $F(u)=u^2(u-1{)}^2$, and

[ \lambda(x)=\begin{cases}\lambda_{0}&\Omega\setminus D\ 0&D\end{cases} ]

is the characteristic function of $\Omega \setminus D$ multiplied by a constant ${\lambda }_0>>1$.

A generalization of the Cahn-Hilliard inpainting approach to an approach for grayvalue images provides the so called $TV-H^{-1}-$ inpainting, see [1]. This is realized by using subgradients of the total variation functional within the flow, which leads to structure inpainting with smooth curvature of level sets. The connection to Cahn-Hilliard inpainting is that solutions of a discretized Cahn-Hilliard inpainting approach $\Gamma$-converge to solutions of an iterative scheme for $TV-H^{-1}$ inpainting. A similar form of this $\Gamma$-limit already appeared in the context of decomposition and restoration for grayvalue images, see for example [18] and [15].

$TV-H^{-1}$ inpainting is proposed in the following way: Let $f\in L^2(\Omega )$, $|f|\le 1$ be the given grayvalue image. The inpainted version $u(x)$ of $f(x)$ evolves in time like

$u_t=-\Delta (\nabla \cdot (\frac{\nabla u}{|\nabla u|}))+\lambda (f-u),$ (2)

where $\hat{p}\in \partial TV(u)$ is replaced by the formal expression $\nabla \cdot (\frac{\nabla u}{|\nabla u|})$.

Both inpainting algorithms (1) and (2) can be numerically solved in a fast way by so called convexity splitting methods, see section 4 and [5] for details.

As mentioned in section 2 the Neidhart frescoes pose a special challenge concerning their digital restoration. We summarize the main issues in the following list:

1. Lack of grayvalue contrast
2. Low color saturation and hue
3. Damaged parts can be rather big, i.e., the diameter of the damaged domain can be larger than the width of lines which are to be continued into the damaged part

So we need an inpainting approach which takes into account these possible difficulties and solves (or circumvent) them. As we have pointed out earlier in this section the third issue can be solved by using a higher order inpainting method such as (1) and (2). Unfortunately difficulties two and three prevent the effective application of these methods. As the contrast between grayvalues is low the edges (which identify the main structure of an image) are not clearly defined. As inpainting lives and dies with uniqueness of edge continuation (cf. Figure 2) we possibly run into troubles if we do not preprocess the digital images of the fresco in an adequate way.

Namely we follow two strategies

Figure 2: (l.) What is the right solution? (r.) Part of the Neidhart fresco: How should the inpainting algorithm decide in this case?

Figure 3: Part of the Neidhart frescoes

• Strategy1: Structure inpainting on binary images with the Cahn-Hilliard equation. Based on the so recovered binary structure the fresco is colorized (cf. [13])
• Strategy2: Apply $TV-H^{-1}$ inpainting in two steps. First with a small ${\lambda }_0$ , e.g., ${\lambda }_0=1$ , to merge together fine artifacts in the fresco by diffusion. In the second step we choose a large \lambda_0 >> 1 , e.g., ${\lambda }_0=10^3$ , to reconstruct the fresco inside the damaged parts.

In the next section we present first numerical results following these two strategies.

4 Numerical Results

In the following numerical results for the two inpainting approaches (1) and (2) are applied to the Neidhart frescoes. For both approaches we used convexity splitting algorithms, proposed by Eyre in [12], for the discretization in time. For more details to the application of convexity splitting algorithms in higher order inpainting compare [5].

For the space discretization we used the cosine transform to compute the finite differences for the derivatives in a fast way and to preserve the Neumann boundary conditions in our inpainting approaches (also cf. [5] for a detailed description).

Both algorithms are applied to small parts of the Neidhart fresco (cf. Figure 4 for orientation) following our two strategies described in section 3.

Figure 4: (l.) Part of the fresco; (m.) binary selection; (r.) Cahn-Hilliard inpainting with ${\lambda }_0=10^7$: f.l.t.r.: binarized selection of the fresco; initial condition for the inpainting algorithm where the inpainting region is marked with a gray rectangle; inpainting result after 200 timesteps with $\epsilon =3$; inpainting result after additional 800 timesteps with $\epsilon =0.01$

Figure 5: Part of the fresco; (m.) binary selection; Cahn-Hilliard inpainting with ${\lambda }_0=10^6$: f.l.t.r.: binarized selection of the fresco; initial condition for the inpainting algorithm where the inpainting region is marked with a gray rectangle; inpainting result after 200 timesteps with $\epsilon =4$; inpainting result after additional 800 timesteps with $\epsilon =0.01$

4.1 Binary fresco inpainting

For the discretization in time we use a convexity splitting scheme applied by Bertozzi et al. [4] to Cahn-Hilliard inpainting. The numerical scheme is of the form

with constants C_1 > \frac{1}{\varepsilon}, C_2 > \lambda_0.

As described in the previous section in strategy 1 we begin with the inpainting of the binary structure of the frescoes by means of (3), cf. Figure 4-5. In our numerical examples we applied (3) in two steps (cf. [4]). In the first few iterations we solve (3) with a rather big $\epsilon$, e.g., $\epsilon =3$. We stop when we are sufficiently close to a steady state. Then we switch the $\epsilon$ value to a smaller one, e.g., $\epsilon =0.01$. Using the steady state from the first few iterations of (3) with a large $\epsilon$ as initial condition, we apply the iteration now for the switched $\epsilon$. Again we stop when we are at, i.e., sufficiently close to, the steady state.

The next step will be to recolorize the damaged parts by using the recovered binary structure as underlying information. This can be done for instance as in [13] and [14].

Figure 6: $TV-H^{-1}$ -inpainting applied to a part of the Neidhart fresco

Figure 7: $TV-H^{-1}$ -inpainting following strategy 2. f.l.t.r. part of the Neidhart fresco; preprocessed image; initial condition for the inpainting algorithm where the inpainting domain is marked as a gray rectangle; preliminary inpainting result (algorithm carried out until $err\approx 10^{-4}$ ). Inpainting difficulties due to the reasons indicated in Figure 2 are clearly visible.

4.2 Grayvalue fresco inpainting

We consider the steepest descent equation (2) and apply as before a convexity splitting method for its time discretization (cf. [5]). The resulting time-stepping scheme is

In order to make the scheme unconditionally stable, the constants $C_1$ and $C_2$ have to be chosen such that C_1 > \frac{1}{\delta} and C_2 > \lambda_0 .

In Figure 6 the algorithm (4) has been applied to a small part of the Neidhart frescoes. In this particular case we didn’t even have to preprocess the image because only plain grayvalue information was to be imported into the inpainting domain. Whereas in Figure 7 we acted on strategy 2. Namely we primarily denoised the image by (4) with $\lambda (x)={\lambda }_0$ on the whole image domain and applied the inpainting algorithm ((4) with $\lambda (x)=0$ inside the inpainting domain $D$ ) on the “cleaned” image in a second step.

Acknowledgments

The authors thank the Applied Partial Differential Equations Research Group of the Department of Applied Mathematics and Theoretical Physics, Cambridge University, the fruitful discussions during the late preparation of this work. M. Fornasier thanks Ingrid Daubechies and the Program in Applied and Computational Mathematics, Princeton University, for the hospitality, during the early preparation

169

of this work. M. Fornasier acknowledges the financial support provided by the European Union’s Human Potential Programme under contract MOIF-CT-2006-039438. C.-B. Schönlieb acknowledges the financial support provided by the Wissenschaftskolleg (Graduiertenkolleg, Ph.D. program) of the Faculty for Mathematics at the University of Vienna, supported by the Austrian Science Fund and of the project WWTF Five senses-Call 2006, Mathematical Methods for Image Analysis and Processing in the Visual Arts. Peter Markowich also acknowledges support from the Royal Society as Wolfson Research Merit Award Holder.

References

• [1] M. Burger, L. He, P. Markowich, C. Schönlieb, Cahn-Hilliard inpainting and a generalization for grayvalue images, in preparation.
• [2] M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester, Image inpainting, Computer Graphics, SIGGRAPH 2000, July, 2000.
• [3] A. Bertozzi, S. Esedoglu, and A. Gillette, Inpainting of Binary Images Using the Cahn-Hilliard Equation, to appear in IEEE Trans. Image Proc.
• [4] A. Bertozzi, S. Esedoglu, and A. Gillette, Analysis of a two-scale Cahn-Hilliard model for image inpainting, submitted to Multiscale Modeling and Simulation.
• [5] A. Bertozzi, C.-B. Schönlieb, Unconditionally stable schemes for higher order inpainting, in preparation.
• [6] V. Caselles, J.-M. Morel, and C. Sbert, An axiomatic approach to image interpolation, IEEE Trans. Image Processing, 7(3):376–386, 1998.
• [7] T.F. Chan, S.H. Kang, and J. Shen, Euler’s elastica and curvature-based inpainting, SIAM J. Appl. Math., Vol. 63, Nr.2, pp.564–592, 2002.
• [8] T. F. Chan and J. Shen, Mathematical models for local non-texture inpaintings, SIAM J. Appl. Math., 62(3):1019–1043, 2001.
• [9] T. F. Chan and J. Shen, Non-texture inpainting by curvature driven diffusions (CDD), J. Visual Comm. Image Rep., 12(4):436–449, 2001.
• [10] T. F. Chan and J. Shen, Variational restoration of non-flat image features: models and algorithms, SIAM J. Appl. Math., 61(4):1338–1361, 2001.
• [11] S. Esedoglu and J. Shen, Digital inpainting based on the Mumford-Shah-Euler image model, UCLA CAM Report 2001-24 at: www.math.ucla.edu/imagers; European J. Appl. Math., in press, 2002.
• [12] D. Eyre, An Unconditionally Stable One-Step Scheme for Gradient Systems, Jun. 1998, unpublished.
• [13] M. Fornasier, Nonlinear projection recovery in digital inpainting for color image restoration, J. Math. Imaging Vis. Vol. 24, No. 3, pp. 359-373, 2006.
• [14] M. Fornasier, and R. March, Restoration of color images by vector valued BV functions and variational calculus, SIAM J. Appl. Math., Vol. 68 No. 2, pp. 437-460, 2007.
• [15] L. Lieu and L. Vese, Image restoration and decompostion via bounded total variation and negative Hilbert-Sobolev spaces, UCLA CAM Report 05-33, May 2005.
• [16] O. M. Lysaker and Xue-C. Tai, Iterative image restoration combining total variation minimization and a second-order functional, International Journal of Computer Vision, 66(1):518, 2006.
• [17] S. Masnou and J.-M. Morel, Level-lines based disocclusion, Proceedings of 5th IEEE Intl Conf. on Image Process., Chicago, 3:259–263, 1998.
• [18] S. Osher, A. Sole, and L. Vese. Image decomposition and restoration using total variation minimization and the H -1 norm, UCLA CAM Report, 02(57), 2002.
• [19] L. Rudin and S. Osher, Total variation based image restoration with free local constraints, Proc. 1st IEEE ICIP, 1:31–35, 1994.
• [20] L.I. Rudin, S. Osher, and E. Fatemi, Nonlinear total variation based noise removal algorithms, Physica D, Vol. 60, Nr.1-4, pp.259-268, 1992.
• [21] A. Tsai, Jr. A. Yezzi, and A. S. Willsky, Curve evolution implementation of the Mumford-Shah functional for image segmentation, denoising, interpolation and magnification, IEEE Trans. Image Process., 10(8):1169–1186, 2001.