Image Stitching: From Mathematics to Arts

Year: 2008 Authors: Franz Schubert; Arjan Kuijper; Mike Kostner

Core claim

Deliberately deviating from stitching assumptions creates new artistic effects, including deconstruction, ghosting, and synthetic reconstruction.

Topics

image stitching, panoramas, artistic deviation, feature matching, visual distortion

Domains

computer vision, scale-space theory, SIFT features, geometric transformation, digital art, photographic panorama, visual effects, constructivism

Methods

automatic stitching, feature detection, blending, controlled assumption violation

Media

digital photographs, analogue photographs, consumer camera, stitching software

Paper text

The text below is the locally extracted OCR/Markdown version of the paper. Raw PDF files remain local and are not published here.

401

Image Stitching: From Mathematics to Arts

Franz Schubert University of applied arts Vienna, Austria

Arjan Kuijper RICAM* Linz, Austria

Mike Kostner University of applied arts Vienna, Austria

Abstract

Image Stitching is concerned with merging several images capturing a scene such that a larger image is obtained. Nowadays, several software packages are available that perform this task automatically. Firstly we discuss the background of one of the most successful ones. Secondly, we point out that the underlying principles of such packages can serve as inspiration for artistic use by using input images that partially deviate from these assumptions.

1 Introduction

Automatically stitching two images capturing different parts of a scene, but partially overlapping, requires a method that can identify the overlapping region, even if the images are made under different conditions, like illumination, angle, occlusions, size, etc. Recently, a novel method was introduced: Autostitch. On the web site if this program (http://www.cs.ubc.ca/~mbrown/autostitch/autostitch.html) one can read:

Autostitch™ is the world’s first fully automatic 2D image stitcher. Capable of stitching full view panoramas without any user input whatsoever, Autostitch is a breakthrough technology for panoramic photography, VR and visualisation applications. This is the first solution to stitch any panorama completely automatically, whether 1D (horizontal) or 2D (horizontal and vertical).

In contrast to several commercial products, the underlying mathematical theory is known. In the following, we discuss this background of this method and show the artistic possibilities of a controlled deviation from the underlying assumptions.

2 Background of Autostitch

One way to characterise an image is by using certain features, like edges, corners, junctions, etc. To overcome the problem of different imaging conditions as much as possible, features should be chosen such that they are invariant to them. Lowe’s SIFT (Scale Invariant Feature Transform) features [4] satisfy this property. They are obtained by computing differential invariants at various scales and their maximal response.

This approach transforms an image into a large collection of local feature vectors, each of which is invariant to image translation, scaling, and rotation, and partially invariant to illumination changes and affine or 3D projection [4].

A scale space [2, 3] is constructed by convolving the image with Gaussian filters at increasing sizes. The difference-of-Gaussians is a scale and rotation invariant image feature detector [3]. The set of most stable extrema of these points in location and scale are taken, together with their orientation and gradient magnitude, and stored in the vector.

Obviously, feature points are located throughout the whole image. To find two clusters of similar SIFT features in two images, the optimal transformation of a number of random subsets of features in one image to those in the other image is computed [1]. The final outcome gives the two subsets in both images that can be matched at minimal costs, i.e. that are closest.

*Johann Radon Institute for Computational and Applied Mathematics (RICAM) / Austrian Academy of Sciences

Figure 1: To stitch a collection of images pairwise, first their SIFT features are computed, then the ones appearing in both images are obtained. The optimal transformation for mapping these points is applied to the second image, yielding partially overlapping images. Blending is used to optimise the colours. Figures taken from the Autostitch web site.

An example is given in Fig. 1 (taken from the Autostitch web site), where a collection of natural images are aligned and blended to form one large stitched image. In the examples shown, Autostitch is mainly used for stitching natural, static images without any moving or changing components.

3 Artistic use of stitching programs

The artistic aim of this paper is to show some artistic approaches in working with stitching software by deviating partially from the underlying assumptions in a controlled manner. The intend is to work with this kind of software to create seamless panoramas from a range of pictures shot with analogue or digital camera hardware in combination with special tripods developed for rotating a camera around its nodal point (the optical point from which the focal length is measured). It aims to create pictures with an extremely wide field of view, often a full $360^{\circ }$ horizontal and $180^{\circ }$ vertical rotation around a fixed centre without any losses in details with a maximum depth of field.

To create accurate panoramas it is necessary to deal with high-end cameras and tripods. None of these tools are used to capture the panoramic views or stitches which are presented on the following pages. As it is possible to create panoramas with stitching software (some of the best are freeware) the authors tried to go beyond the limits of it - one challenge was to use a low-price digital consumer still camera - and to distinguish some series from rough shots with the camera free moving using a simple tripod for more exact rotations (note: the camera was not rotated exactly around its nodal point). Consecutive single shots should overlap each other at an area of approximately $30\%$ to get the best result with digital stitching software.

4 Experiments

The artistic experiments are structured in 4 main sectors:

1. Panoramic views of interior and exterior environments, often influenced and disturbed by a lack of details and therefore insufficient unique matching points for panorama alignment. This results in destructed and / or reconstructed representations of the three dimensional environment.

2. Panoramas gathered from footage of a camera not fixed into place (series of pictures from a moving camera are not suitable for stitching panoramas. In order to correctly stitch images together without parallax error, the camera needs to be rotated about its nodal point). Here the spatial relation between feature points of foreground objects changes, yielding different matching possibilities.

3. Stitched (or segmented) panoramas presenting objects / persons moving between each single shot. Like other photographic multiple exposure images these panoramas multiply the appearance of these objects / person in the resulting panorama. If they are situated in overlapping areas they appear ghosted / transparent.

4. Panoramas of objects/persons gathered by orbiting the camera around and taking consecutive pictures. The object is often twisted and deformed like in works of synthetic cubism. The background provides sufficient feature points for stitching, but the object may introduce special effects in the final stitched image.

All the digitally stitched panoramas shown in this paper have one thing in common: their background environment features are used by the stitching software to compose a three dimensional space which acts as reference for reordering a series of photographs. The representation of the main subject (object / person in the centre of view) is influenced by this process, there is an automatic re-assignment of its parts, views and sometimes of the content itself. For page constraints, only single examples are shown for two cases. More examples are available at http://www.ricam.oeaw.ac.at/people/page/kuijper/stitch.pdf.

4.1 Moving camera

An first example consists of stitching a series of 33 pictures (see Figure 2), taken when walking by a street in Vienna, with an area overlap of about $30\%$ . The images are captured without a tripod and with a moving camera. The automatic stitching process favourites plane, even objects in the background. Objects fully articulated in three dimensions are blended against this background. Parallax errors are the main reason of ghosting foreground objects (cars).

Figure 2: Moving camera: Stitching a series of 33 pictures captured outdoor. Top row: AutoStitch V2.187. Bottom row: Adobe Photoshop CS3 (Manual Position/Rotation Adjustment, Automerge)

4.2 Moving objects

A second example considers moving objects. In Figure 3 we stitched 21 pictures, taken with the camera mounted on a tripod in one horizontal line. There is one person in front of the camera on every single picture. The results of the different programs can be seen in Figure 3. The red ellipses show significant differences between the programs.

Figure 3: Moving objects. Top row: AutoStitch V2.187. Middle row: PTGui V 2.71 Pro. Bottom row: Adobe Photoshop CS3 (Photomerge Zylindrical).

5 Discussion

The automatic image stitching programs assume that the input images are more or less obtained from natural scenes. This implies that the images contain enough diversity, and consequently enough (SIFT) feature points. Furthermore, the spatial relations between the points need to be rather fixed.

Therefore, constructing a series of images that violates this assumptions partially and in a controlled manner will construct a large stitched image that does not represent the real world, but an artistic decision which refers to the process of constructivism in the realm of object representation.

Secondly, for stitching it is mostly assumed that the camera position is more or less constant. The street sequences violate this assumption for foreground object, but the background provides sufficient information for proper stitching. This yields possibilities for creating illusory effects of foreground objects.

Finally, the presence of an object (subject) moving with the camera does not disturb the correct stitching of the background. This is expected as most feature points arise from this background. However, the blending of two merged images creates a ghostly effect when the object is within the region of overlap. As we noticed, different software packages use different types of blending (not always documented) and give therefore different effects.

This survey of image stitching programs shows that explicitly deviating from the principles of the underlying mathematical model in a controlled manner creates new possibilities in the visual arts. Our aim was to deviate only partially from the assumptions to create illusory effect, sometimes showing the range from deconstruction to synthetic reconstruction of visual object representations.

References

[1] M. Brown and D.G. Lowe. Automatic panoramic image stitching using invariant features. International Journal of Computer Vision, 74(1):59-73, 2007. [2] L. M. J. Florack. Image Structure, volume 10 of Computational Imaging and Vision Series. Kluwer Academic Publishers, Dordrecht, The Netherlands, 1997. [3] T. Lindeberg. Scale-Space Theory in Computer Vision. The Kluwer International Series in Engineering and Computer Science. Kluwer Academic Publishers, 1994. [4] D.G. Lowe. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2):91-110, 2007.