CollageBild: A System to Create Image Mosaics

Year: 2023 Authors: André R. S. Marcal

Core claim

The system produces collage mosaics effectively, but the best outcome depends on parameter choices, reference-image diversity, and visual aesthetics.

Topics

image mosaics, collage generation, color distance, aesthetic parameter tuning

Domains

digital image processing, color spaces, distance metrics, sampling grids, generative art, mosaic design, visual composition, aesthetic evaluation

Methods

image partitioning, reference pre-processing, pixel comparison, spatial tolerance

Media

Matlab, RGB images, reference image sets, synthetic image references

Paper text

The text below is the locally extracted OCR/Markdown version of the paper. Raw PDF files remain local and are not published here.

Bridges 2023 Conference Proceedings

CollageBild: A System to Create Image Mosaics

André R. S. Marcal

Departamento de Matemática, Faculdade de Ciências, Universidade do Porto, Portugal andre.marcal@fc.up.pt

Abstract

This paper presents a system, called CollageBild, to create an image mosaic (or collage) by putting together smaller images. The minimum requirement is to have a base image, a set of reference images, and a value for the minimum number of blocks along row or column. Several additional processing parameters are used to control the color model, color distance, and image sampling strategy. The system is described and applied to test cases with a variety of real and synthetic references. The results vary considerably, not only with the parameters setting, but also with the reference images used. There is no ideal result, as the decision of what is the best setting is based on aesthetics.

Introduction

An image can be constructed by small blocks that are imperceptible when the image is viewed from a distance. This is in fact what happens in a digital image, where each image pixel is a square block, and on a computer screen or TV, where the individual light-emitting elements are very small. Usually the goal is to have these individual elements that form the image unnoticed. However, in some contexts, it can be interesting to use rather large blocks or to have them clearly visible, for example to create a tapestry, an image using building tiles, or an image mosaic (sometimes referred to as collage) composed of individual images that when observed from a distance form another image.

This paper presents the CollageBild system that can be used to create image mosaics or collages, and additionally to explore relations between human color perception and color models and distances. The CollageBild system was developed in Matlab [6] and is available for public use at [5]. The system has a number of controlling parameters that will be described next, together with several application examples.

The minimum requirement is to have a base image, a set of reference images, and a value for the minimum number of blocks along a row or column $(N)$ . To simplify the description, an initial example is presented in Figure 1, using a sunflower as the base image (Figure 1(a)). Three results produced by CollageBild are presented in Figure 1 for different values of $N$ : 8, 12 and 25. The reference images used here were Lego blocks (Figure 3(a)), described in detail later. As $N$ increases, the image blocks become less prominent, and the details of the base image are better represented. However, if $N$ is too large, the perception of the individual blocks is lost and there are thus no benefits in having a collage instead of the base image.

(a)

(b)

(c) Figure 1: Base image (a) and CollageBild results with Lego blocks for: (b) $N=8$ , (c) $N=12$ , (d) $N=25$ .

(d)

Marcal

Method

The processing involved in creating a collage include an initial stage where the base image is partitioned into sub-images, a reference pre-processing stage, and the sequential analysis of each individual sub-image.

The list of parameters used by the CollageBild system are: base image, reference image set, $N$ , color model/metric, spatial tolerance, number of elements to use for the sampling grid $(N_S)$ .

Image Partition

The initial step is to compute the number of blocks (or sub-images), along rows and columns, to be extracted from the base image. This takes into account the base image size, the reference images row-to-column ratio, and the value of $N$ . It is computed to maximise the block size. For example, the ‘Sunflower’ base image (Figure 2(a)) has $559\times 710$ pixels, and the Lego reference image set used to create the collage images of Figure 1 have $31\times 31$ pixels; thus a row-to-column ratio of 1. Using $N=8$ results in a block size of $69\times 69$ pixels ( $559/8=69.88$ ), and a total of 10 blocks in columns ( $710/69=10.29$ ). This grid of $8\times 10$ blocks corresponds to $552\times 690$ pixels; thus the grid does not cover the whole base image. The grid is placed in the centre of the base image (Figure 2(b) with, in this case, 7 rows and 20 columns discarded from the base image. For $N=12$ the grid has $12\times 15$ blocks of $46\times 46$ pixels (Figure 2(c)). For $N=25$ the grid has $25\times 32$ blocks of $22\times 22$ pixels (Figure 2(d)), which in this case are smaller than the reference images.

(a)

(b)

(c) Figure 2: ‘Sunflower’ base image (a) with block regions overlaid in red for: (b) $N=8$ , (c) $N=12$ , (d) $N=25$ .

(d)

Sampling and Reference Pre-Processing

Once the size of the base image blocks is known ( $N_r^B\times N_c^B$ pixels), together with the size of the reference images ( $N_r^R\times N_c^R$ pixels), the next step is to compute the actual number of sampling points ( $N_s^{\prime }\times N_S^{\prime }$ ) that will be used to compare each sub-image with the references. The value is $N_S^{\prime }=\min \{N_S,N_r^B,N_c^B,N_r^R,N_c^R\}$ .

As a pre-processing stage, if N_S' < \min\{N_r^R, N_c^R\} , a smaller sized reference set is prepared to be used for comparison with the image blocks. The resulting image set is a 4D matrix with $N_S^{\prime }\times N_S^{\prime }\times 3\times N_R$ elements, where $N_R$ is the number of reference images and 3 corresponds to the image color components (R,G,B).

Image Comparison

The selection of the most suitable reference image to be matched with a base image block is done by comparing the color of the individual pixels of the block, extracted in a $N_S^{\prime }\times N_S^{\prime }$ grid, with each reference. The decision criteria is to select the reference which has the lowest average color distance to the block.

There are several color models and metrics that can be used to compute the color distance between two image pixels. The most widely used color model in digital image processing is probably RGB [2], where the three variables red (R), green (G) and blue (B) are the 3 primary colors of this additive model. In fact, both the base image and the reference images are expected to be in a RGB standard format (24-bits). However, unlike RGB, other color models have the color component decoupled from the intensity, which can be beneficial for color analysis, such as the HSV (Hue, Saturation, Value) model [4], and the CIELAB color space, also referred to as Lab (or $L^{\ast }{a}^{\ast }{b}^{\ast }$ ) model [7].

CollageBild: A System to Create Image Mosaics

The color distances $(d)$ implemented in CollageBild, all in a normalised form with the co-domain [0, 1], are: (1) Euclidean RGB, (2) City Block RGB, (3) adjusted City Block HSV, (4) Euclidean Lab, (5) City Block Lab, (6) Hybrid Lab [1].

The evaluation of the similarity between two images is based on the mean normalised color distance between all image pixel pairs. However, the direct pairing of pixels from two images might not be the most suitable approach, as there might be a small geometric misalignment between the images. It is thus worth considering a small tolerance in what regards the positioning of the pixel pairing. To address this issue, a spatially tolerant color distance $D^{(\nu )}$ is used, considering a neighborhood $\nu$ of a pixel. It is defined by (1), where each pixel $(x,y)$ in image $I_1$ is compared with all pixels $(x^{\prime },y^{\prime })$ in image $I_2$ that are within the neighborhood $\nu$ of $(x,y)$ . The color distance $D^{(\nu )}$ used is the smallest distance of all $(x,y)-(x^{\prime },y^{\prime })$ pairs.

The neighborhoods used here are: direct pairing $(\nu =1)$ , where $x=x^{\prime }$ and $y=y^{\prime }$ ; 4-neighbors $(\nu =4)$ , where $|x-x^{\prime }|+|y-y^{\prime }|\in \{0,1\}$ ; 8-neighbors $(\nu =8)$ , where $x-x^{\prime }\in \{-1,0,1\}$ and $y-y^{\prime }\in \{-1,0,1\}$ . A total of 18 color distances are thus considered: 6 distances/models $\times$ 3 neighborhoods $(\nu =1,\nu =4,\nu =8)$ .

Reference Image Sets

The CollageBild system was developed in a way that any set of reference images can be used, which nevertheless need to be RGB 24-bit images, all of the same size (number of rows and columns). The reference sets that were used in this paper are described next, with a summary presented in Table 1.

Table 1: Summary of references image sets used.

The first one is the Lego blocks set (‘Lego’), which was used to create the mosaics of Figure 1. The 18 individual reference images, all $31\times 31$ pixels, are presented in Figure 3(a). These images have a circular region with a near uniform color, over a dark background, with some bright spots due to illumination and reflection effects. Another reference set with similar characteristics is ‘M&Ms’, presented in Figure 3(b). It has 22 images of $55\times 55$ pixels. The sampling of both of these reference sets should be done with a grid of $2\times 2$ ( $N_S=2$ ) to avoid sampling the background. The ‘Color Tiles’ reference set has 25 square color tiles, in $69\times 69$ pixel images (not shown). They also have some bright pixels due to specular reflection, but no background, as the mosaics are square shaped. As all these reference images are roughly isotropic, with nearly constant color for all points sampled, the neighborhood is irrelevant (thus $\nu =1$ is recommended).

(a)

(b) Figure 3: Two of the reference sets: (a) ‘Lego’, with 18 elements; (b) ‘M&Ms’, with 22 elements.

Marcal

The ‘Azulejos’ (decorative tiles) reference set has 112 rather large images ( $267\times 267$ pixels), with diverse colors and patterns. For these images, due to their diversity, it is worth using a large value for $N_S$ and a spatially tolerant color distance ( $\nu =4$ or $\nu =8$ ).

Most of the ‘Azulejos’ images are unchanged by rotation. However, for those that are not, it is worth considering some additional reference images obtained by rotation. Figure 4 shows two examples: one where the original image (Figure 4(a)) can provide 3 additional versions, by rotations of $90^{\circ }$ , $180^{\circ }$ and $270^{\circ }$ (Figures 4(b), 4(c), 4(d)); and another (Figure 4(e)) where only 1 rotation of $90^{\circ }$ is relevant (Figure 4(f)).

(a)

(b)

(c)

(d)

(e) Figure 4: Examples of ‘Azulejos’ that require rotation: $(a,e)$ original; $(b,f)$ $90^{\circ }$ , $(c)$ $180^{\circ }$ , $(d)$ $270^{\circ }$ rotation.

(f)

Synthetic Modifications

Some additional modifications in the reference images can be done, as long as the aim is simply to produce a digital image collage. The permutation of the 3 color components (R,G,B) is a possible approach. As an example, Figure 5 shows an original image from the ‘Azulejos’ reference set (Figure 5(a)) and the 5 possible RGB color composite permutations: 132, 213, 231, 312, 321 (Figures 5(b)-5(f)). A further extension is possible by including also the intensity inverted R,G,B color components. Using both the inversions and permutations, a total of 48 color images are created from the original RGB image. Figure 6 shows 24 of these synthetic images, created for the original version on the top left corner.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 5: RGB color composite permutations: (a) original 123, (b) 132, (c) 213, (d) 231, (e) 312, (f) 321. Figure 6: Synthetic images created by inversions and permutations of the RGB color components.

CollageBild: A System to Create Image Mosaics

The reference set ‘Azulejos Inv+Perm’ is based on the permutations and inversions procedures described for a single RGB image, as illustrated in Figure 6, and it thus have 48 elements. The reference set ‘Azulejos Extended’ contains the original 112 ‘Azulejos’ images together with some additional images created by rotations and RGB permutations with 256 elements in total.

Results

The CollageBild system outputs not only the collage image but also the mean color distance $(\overline{D})$ between the base image blocks and the references selected. This distance can be used to evaluate how well a set of references matches the base image. However, the lowest distance does not always correspond to the most aesthetically pleasing result. The various color models and distances also provide different results. Although the Lab based distances are more closely aligned with the human color perception, the results provided by these metrics are not always the most interesting ones. In fact there is not an ideal or best result.

Figure 7 shows an example of CollageBild results for multiple references, using a base image of the Peggy’s Cove Lighthouse (Figure 7(a)), with $N=22$ and the Lab Euclidean distance. The references and mean color distances are: ‘Lego’, $\overline{D}=0.038$ (Figure 7(b)); ‘M&Ms’, $\overline{D}=0.049$ (Figure 7(c)); ‘Color Tiles’, $\overline{D}=0.044$ (Figure 7(d)); ‘Azulejos’, $\overline{D}=0.047$ (Figure 7(e)). For this particular image, the result produced with the ‘Color Tiles’ reference set is perhaps the most interesting.

(a)

(b)

(c)

(d)

(e) Figure 7: CollageBild results for $N=22$ , Lab Euclidean distance, and multiple references: (a) base image, (b) ‘Lego’, (c) ‘M&Ms’, (d) ‘Color Tiles’, (d) ‘Azulejos’.

Figure 8 show some additional results with the same base image, with $N=22$ and the ‘Color Tiles’ references, for the other color distances: RGB Euclidean (Figure 8(a)), RGB City Block (Figure 8(b)), HSV City Block (Figure 8(c)), Lab City Block (Figure 8(d)), Lab Hybrid (Figure 8(e)).

(a)

(b)

(c)

(d)

(e) Figure 8: CollageBild results for $N=22$ , ‘Color Tiles’ references, and multiple distances: (a) RGB Euclidean, (b) RGB City Block, (c) HSV City Block, (d) Lab City Block, (e) Lab Hybrid.

Marcal

A final example is presented in Figure 9 using a photograph of Reza Sarhangi, the founder of the Bridges Organization [3], as base image (Figure 9(a)). Several sets of parameters were tested, but only two results are presented. The first one is based on the Lego reference set (Figure 9(b)), with $N=40$ , RGB City Block distance, $N_S=2$ and $\nu =1$ . The second is based on the ‘Azulejos Inv+Perm’ references (Figure 9(c)), using the RGB image of Figure 4(a) as the source, with $N=35$ , RGB Euclidean distance, $N_S=5$ and $\nu =8$ .

(a)

(b) Figure 9: CollageBild results for Reza Sarhangi: (a) base image [3], (b) with ‘Lego’ references, (c) with ‘Azulejos Inv+Perm’ references.

(c)

Summary and Conclusions

The CollageBild system presented is effective in producing an image mosaic, or collage, by putting together smaller images from a reference set. The quality of the final result, aesthetically, depends not only on the parameter settings used, but also on the reference set. These images should cover a range of colors with some diversity, but not too different from the predominant colors on the base image. The method described to create synthetic images can be used to fill some gaps in the reference set, if the goal is to have a digital collage only. As the visual perception of the collage is strongly dependent on the distance of the observer, it is useful to try many different versions to identify the most satisfactory result. CollageBild is well suited for this, as it allows for multiple configurations to be tested easily. CollageBild is available for public use at [5].

References

[1] S. Abasi, A. T. M., and M. Fairchild. “Distance Metrics for Very Large Color Differences.” Color Research & Application, vol. 45, 2020, pp. 208-223. [2] M. Bratkova, S. Boulos, and P. Shirley. “oRGB: A Practical Opponent Color Space for Computer Graphics.” IEEE Computer Graphics and Applications, vol. 29(1), 2009, pp. 42-55. [3] Bridges. Reza Sarhangi. https://www.bridgesmathart.org/reza-sarhangi/. [4] R. Gonzalez and R. Woods. Digital Image Processing, 3rd ed. Pearson, 2008. [5] A. Marcal. André Marçal web page, Software. https://www.fc.up.pt/pessoas/andre.marcal/page5.. [6] MathWorks. “MATLAB, ver R2021a.” 2021. MATLAB Central File Exchange. Retrieved September 7, 2022. www.mathworks.com. [7] E. Reinhard and T. Pouli. “Colour Spaces for Colour Transfer.” Lecture Notes in Computer Science, vol. 6626, 2011, pp. 1-15.