Image Stitching Method of Non-Ferrous Smelting Scene Based on SIFT Algorithm and Color Constancy

Abstract

In non-ferrous smelting scenes, complex environments, uneven lighting conditions, and inaccurate automatic exposure parameters result in the panoramic image having obvious seams and significant brightness differences in different regions. To address these issues, a novel image stitching method based on Scale-Invariant Feature Transform (SIFT) and color constancy theory was proposed. Initially, the input image was preprocessed followed by feature extraction and matching. Subsequently, color constancy processing based on matching points was performed to acquire an image with consistent brightness to be spliced. This process was complemented by the integration of multi-band fusion to enhance the original fusion procedure. Eventually, a spliced image with seamless blending and even luminosity was generated. The method proposed in this paper can obtain panoramic images that were more suitable for human eyes to observe, and greatly improve the subjective and objective performance of panoramic images. In the non-ferrous smelting scene, the PSNR and SSIM scores were improved by 3.555 dB and 0.16 respectively.

Share and Cite:

Liu, Y. and Long, Y. (2023) Image Stitching Method of Non-Ferrous Smelting Scene Based on SIFT Algorithm and Color Constancy. Open Journal of Applied Sciences, 13, 2055-2067. doi: 10.4236/ojapps.2023.1311161.

1. Introduction

Non-ferrous metallurgy is an important strategic industry for the development of national defense and the promotion of related scientific progress. Usually, the industrial scene of non-ferrous is large in scale and complex in an environment that requires large-scale video surveillance. Limited to the narrow view field of common camera the captured images cannot meet the monitoring requirements. Therefore, in practical applications, image stitching technology for obtaining the corresponding panoramic image has a very important significance.

However, due to the complexity of the scene and the presence of multiple different light sources, the images captured by different camera types of equipment often have significant differences in color and brightness. Directly stitching these images would result in obvious seams and significant differences in brightness between different areas, which would greatly affect subjective perception and subsequent intelligent processing. Existing panorama stitching software in the market, such as AutoStitch, Microsoft’s ICE (Image Composite Editor), and the stitching tool in Photoshop have performed well in image stitching. However, they require the overlapping regions of the images to form a plane and the camera’s optical center to be close to alignment during shooting, which is difficult to achieve in practice. To solve the problems of uneven panoramic image quality, uneven lighting and color difference at the stitching joints, S Yang et al. [1] proposed an optimal multi-scale image stitching fusion method using lighting compensation and new energy functions. Zaragoza [2] proposed the APAP (As-Projective-As-Possible Image Stitching with Moving DLT) algorithm, which divided the image into dense grids and used a homography matrix for each grid to align, while also proposing a moving direct linear transform method to adjust or finesse the pixel warping to adapt to the deviation between the input data and the ideal condition. After the APAP algorithm, X. Pan [3] et al. combine the global two-dimensional homography method with local warping based on mesh optimization to better deal with the parallax problem. Chen [4] proposed a graph-based hypothesis generation and a seam-guided local alignment for improving the effectiveness and the efficiency of the seam-driven approach. Lin [5] proposed an Adaptive As-Natural-As-Possible (AANAP) algorithm to improve the visual perception of stitching results. This algorithm converts the stitching process into a global similarity transformation through linearized homography, which helps alleviate perspective distortions in non-overlapping regions. AANAP is easy to generalize to multiple images and automatically obtains the optimal perspective view in panoramic images.

The existing image mosaic methods cannot solve the problems of obvious seam and uneven brightness and color of panoramic image after the mosaic of nonferrous smelting scene. Through analysis, it is found that the main reason for this problem is that the installation positions of cameras in different positions are too different, and the scene light source is numerous and the environment is complex, which makes the color brightness consistency of images taken in different positions poor, thus affecting the effect of subsequent panoramic mosaic.

To solve this problem, we try image enhancement algorithm based on histogram equalization, image enhancement algorithm based on deep learning, and enhancement algorithm based on Retinex model. The main idea of histogram equalization processing method is to rearrange the histogram of the pixel value distribution in the image by using the cumulative distribution function, so as to make the distribution of pixels in the image as balanced as possible in various color levels, so as to solve the problem that the content of different cameras is too different. Histogram based processing methods can effectively improve the pixel distribution of images, making images taken by different cameras have similar distribution, but this kind of methods cannot solve the problems of missing details and color distortion, and it is difficult to ensure the consistency of brightness when the content of different cameras is too different. The algorithm of brightness constancy based on deep learning has also been greatly developed in recent years. Wei [6] and others proposed retinexnet. On the basis of Retinex theory, convolutional neural network is used to decompose the reflected component and incident component, and build the illumination enhancement network to recover the decomposed incident component. Zhang et al. Proposed kind network [7] according to the idea of “decomposition before recovery”, and added reflection component recovery network to the model accordingly. Although the image enhancement method based on deep learning has achieved good results in some data sets, it depends on the acquisition of paired data. However, in the non-ferrous smelting scene, there is lack of sufficient paired data, and it is difficult to obtain paired data. The enhancement algorithm based on Retinex theory is often used in dark scenes and color offset repair scenes. It restores the original image by removing the influence of the incident component in the image imaging model. The single scale Retinex algorithm [8] uses Gaussian filtering to approximate the incident component, and gets the enhanced image after logarithmic transformation. Retinex theory simulates the physiological characteristics of human eyes and has better color constancy and brightness constancy effects than other transform enhancement algorithms. Among them, automatic color enhancement (ACE) [9] based on Retinex theory has better effects on color brightness uniformity and detail retention.

Experiments show that Retinex, ACE and other color constancy algorithms have good results in solving the problem of uneven illumination, but the original ace algorithm has high computational complexity. For NxN images, the computational complexity is O (N4). Later, Getreuer et al. Proposed a fast ace algorithm, which greatly reduces the complexity of the algorithm by decomposing the slope function and other operations in ace into convolution operations [10] , It makes it possible in practical application. Considering the characteristics of image mosaic algorithm, this paper proposes a non-ferrous smelting mosaic method based on SIFT and color constancy by referring to the traditional color constancy algorithm and combining the related characteristics of multi frame fusion algorithm, and finally obtains the panoramic image with smooth seam and uniform brightness. The main contributions of this paper are as follows:

1) Aiming at the problems of panoramic mosaics in nonferrous smelting scenes, this paper proposes a processing framework based on SIFT and color constancy.

2) Combining the results of feature matching, this paper proposes a color constancy processing method based on matching points, which effectively reduces the brightness difference between different images.

3) Experiments show that in the non-ferrous smelting scenario, the method proposed in this paper has achieved significant benefits both subjectively and objectively, and the PSNR and SSIM indexes have improved by 3.555 dB and 0.16 respectively.

2. Method

As shown in Figure 1, the scheme proposed in this paper can be divided into four modules as a whole, including a preprocessing module, a feature extraction and matching module, a color constancy processing module based on matching points, and an image stitching module.

2.1. Image Preprocessing Module

In the nonferrous smelting scene, because there are multiple light sources in the same space, the image is affected by different light intensities, resulting in the change of gradient value and image pixel value, which will have a great impact on the extraction and matching of key points. To solve this problem, this paper uses the single-scale Retinex algorithm to preprocess the image to improve the accuracy of feature matching.

The single scale Retinex algorithm is based on the separation characteristics of low-frequency and high-frequency information in images by the human visual system, and processes them separately. By adjusting the intensity information of the image to eliminate the influence of lighting, image enhancement is achieved. The basic assumption of this algorithm is that the original image is the product of the illumination image and the reflectance image. After transferring the image to the logarithmic domain during the implementation process, a fixed size Gaussian convolution is used to filter the image:

r ( x , y ) = log ( I ( x , y ) ) log ( I ( x , y ) G ( x , y ) ) (1)

Linear lifting and quantization of the image after filtering:

output ( x , y ) = r ( x , y ) min ( r ) max ( r ) min ( r ) 255 (2)

Figure 1. The process of image stitching of a non-ferrous scene.

The single-scale Retinex algorithm is suitable for processing images with continuous brightness changes, and the calculation speed is relatively fast, which is suitable for image preprocessing. The Gaussian convolution kernel used in this paper is 80. As shown in Figure 2, the result of feature matching after preprocessing increased from 72.76% to 76.70%.

2.2. Image Feature Extraction and Matching Based on SIFT

Due to the complex environment of a nonferrous smelting site, the robustness of feature extraction is required to be high. After comparing the effects of surf, Harris, fast, sift, and other algorithms, this paper selects SIFT algorithm for feature extraction. In this scenario, SIFT algorithm has the following advantages:

Ÿ Contrast robustness. SIFT algorithm obtains stable feature points through differential calculation of Gaussian filters with different sizes, which can reduce the impact of complex illumination changes.

Ÿ Scale invariance. The size, distance, and shape of objects in the smelting scene may change. SIFT algorithm detects local features on different scales of the image and selects the optimal matching points in different focal lengths. It has scale invariance and can better deal with the transformation of different scenes.

Ÿ High precision. Compared with other feature extraction methods, the feature points detected by SIFT are more accurate.

After SIFT feature extraction, the position, scale, main direction, and descriptor of feature points can be obtained, and the descriptor can be used to match the feature points. The feature points after feature matching often have matching errors, so this paper uses the RANSAC algorithm to filter the matched feature points, and the matching effect after filtering is shown in Figure 2. Figure 2 shows the results of RANSAC after removing interior points before and after pretreatment. The green circle represents interior points and the red circle represents exterior points. The matching accuracy before and after pretreatment is 72.76% and 76.70% respectively.

2.3. Color Constancy Processing Based on Matching Points

To further optimize the problem of complex lighting changes in the non-ferrous smelting scene, a color constancy processing method based on matching points is proposed in this paper. The brightness information of matching points extracted by SIFT is fused into the color constancy processing to further reduce the

Figure 2. RANSAC result w/o preprocess VS RANSAC result w/o preprocess.

brightness and color difference between the images to be spliced. The algorithm is shown in Figure 3, which can be divided into three parts, including chromatic/spatial adjustment, image dynamic expansion, and brightness regression based on matching points. The part of chromatic/spatial adjustment and image dynamic expansion refers to the idea of ACE algorithm, which considers the spatial relationship between color and brightness in the image, carries out adaptive filtering of local characteristics, realizes image brightness and color adjustment with local and nonlinear characteristics, and realizes the color constancy and brightness constancy of the image. The correlation between adjacent frames will not be considered due to chromatic/spatial adjustment and image dynamic expansion. In this paper, combined with the brightness information between matching points, the brightness inconsistency between images to be spliced is further optimized by using the inter-frame information. The algorithm flow is shown in Figure 3.

As shown in Figure 3, the image first passes through the color/spatial adjustment module, which refers to the ACE algorithm. By calculating the difference sum between the current pixel and the surrounding pixels and combining it with the spatial distance between pixels, the reconstructed pixels are obtained. The mathematical expression is as follows:

R c ( p ) = j Subset , j p r ( I c ( p ) I c ( j ) ) d ( p , j ) (3)

where, I c ( p ) I c ( j ) refers to the lateral suppression mechanism, that is, the difference between the illuminance of the current pixel and the reference pixel. r is the brightness performance function. d ( p , j ) is the distance between p and j, which is used to represent the weighting of the local and global difference of the pixels. Where r is a saturation function, expressed as:

r ( x ) = { 1 , x < T x /Slope , T x T 1 , x > T (4)

In most cases, it has good color correction effect, which is the critical value. Then the corrected image is dynamically expanded, and the dynamic mapping implementation formula is as follows:

Figure 3. Process of color constancy algorithm based on matching points.

Y c ( p ) = round [ 127.5 + s c R c ( p ) ] (5)

where, s c represents the slope of the line segment [ ( m c , 0 ) , ( M c , 255 ) ] , Y c represents the output image after ACE enhancement.

The core of ACE algorithm is the regional adaptive filtering of image. The larger the scope of action, the better the effect, but the larger the amount of calculation, which limits the practical application. Pair size n * n, In order to reduce the complexity of the algorithm, we use ACE’s fast algorithm [10] to change the global difference to local difference. At the same time, in order to obtain the global information of the image, this paper uses the recursive enhancement in the form of multi-scale pyramid, so that the computational complexity of the algorithm is reduced to o (N2). In order to further reduce the computational complexity, this paper reuses the pyramid structure in SIFT and the pyramid structure in fast ACE through pyramid reuse, so as to further reduce the time complexity of the algorithm. The algorithm flow is shown in Figure 4.

After that, using the correlation between frames, the brightness regression based on matching points is done for the image to be stitched. First, calculate the average brightness difference of matched SIFT features between adjacent frames. The formula is as follows:

O f f s e t = i = 0 N ( B ( f i 1 ) B ( f i 2 ) ) N 1 2 (6)

where, f i 1 , f i 2 represents the i-th group of matching feature points, and B ( * ) represents the brightness value of the calculated feature point area. N represents the number of feature points matched between frames. The image is corrected according to the difference between the statistical features, so as to further reduce the brightness difference between the images to be matched. The formula is as follows:

Y ^ c 2 = Y c 2 O f f s e t (7)

Y ^ c 2 = Y c 2 + O f f s e t (8)

Figure 4. Pyramid reuse of SIFT and fast ACE algorithm.

2.4. Transformation Matrix Extraction and Image Fusion

After obtaining the position information of the matching points and the enhanced image, the MDLT (Moving Direct Linear Transformation) method can be used to calculate the global transformation matrix. For the images to be concatenated I and I’ their corresponding matching points are p = [ x , y , 1 ] and p = [ x , y , 1 ] , the transformation relationship is:

p ~ H p (9)

[ x y 1 ] ~ [ h 11 h 12 h 13 h 21 h 22 h 23 h 31 h 32 h 33 ] [ x y 1 ] (10)

In homogeneous coordinates, it represents proportionally equal, while in non homogeneous coordinates, on non homography surfaces, there are:

x = h 11 x + h 12 y + h 13 h 31 x + h 32 y + h 33 (11)

y = h 21 x + h 22 y + h 23 h 31 x + h 32 y + h 33 (12)

From this, we can obtain:

A h = 0 (13)

where,

A = [ x y 1 0 0 0 x x x y x 0 0 0 x y 1 x y y y y ] (14)

h = [ h 11 h 12 h 13 h 21 h 22 h 23 h 31 h 32 h 33 ] T (15)

When I and I’ have N pairs of matching interior points, the homography matrix of the DLT algorithm can be expressed as:

h ^ = argmin h A h 2 s .t . h = 1 (16)

The obtained homography matrix h ^ can be reconstructed H to solve the problem. After obtaining the local homography matrix, traverse each grid and map it onto the panoramic canvas for image fusion.

Considering that the method of multi band fusion involves processing overlapping regions in all frequency domains, the image quality after stitching will be higher. Therefore, this article selects this fusion method to optimize the stitching process. Multi band fusion adopts a Laplacian pyramid structure. Firstly, the image is decomposed into weighted averages near the overlapping areas of the image in different frequency domains, and then all sub band images at different frequencies are reconstructed into a total image. By determining the size and weight coefficient of the color fusion region based on the differences between the features of two images, it is possible to smoothly transition images with different intensities.

3. Results and Discussion

Compared with the traditional image mosaic method, this paper adds a preprocessing module and a color constancy processing module based on matching points. In this paper, relevant ablation experiments are carried out to verify the effectiveness of the method. At the same time, this method is compared with the existing methods, and the appropriate evaluation index is selected to verify the performance of the splicing algorithm in the non-ferrous smelting scene.

3.1. Image Quality Evaluation Index

At present, the evaluation methods of image mosaic quality can be divided into subjective quality evaluation and objective quality evaluation methods. In this paper, two methods are selected to evaluate the experimental results. The subjective quality evaluation method is to evaluate the image quality according to the subjective perception of the observer on the image, which depends on the human visual system, and the evaluation result is the most reliable. However, there are differences in subjective evaluation, and the evaluation process is complex and time-consuming, which has great limitations in practical application. Objective quality evaluation method is a quantitative analysis of the quality of image mosaic. In image mosaic, it is mainly used to evaluate the accuracy of image registration, that is, to measure the quality of image mosaic by calculating the similarity between images in overlapping areas. Common objective evaluation indexes include root mean square error (MSE) [11] , peak signal-to-noise ratio (PSNR) [12] , structural similarity index (SSIM) [13] , etc.

To sum up, this paper uses a combination of subjective evaluation method and objective evaluation method to evaluate the splicing quality, and the objective evaluation indexes are peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM).

3.2. Experimentation

Experiment 1: with or without color constancy module

In the experimental process, firstly, the image is preprocessed, and sift is used for feature extraction and matching of the preprocessed results. Then, the color constancy processing module based on matching points is used, and finally the image is stitched. Compared with traditional image mosaic methods, this paper adds a preprocessing module and a color constancy processing module based on matching points. The preprocessing module and the color constancy processing module based on matching points are collectively referred to as the color constancy model. In order to verify the effectiveness of the method, we remove the color constancy module, and then compare the mosaic images of the two methods. The results are shown in Figure 5. The above results are the results of removing the color constancy module, and the following results are the results of using the constancy module. It can be seen that the image mosaic result with the color constancy module is clearer, and the details and textures are more obvious. For example, the details of the leftmost area in Figure (c) are shown, and the wall that could not be seen clearly because of the dark light is clearer after the color constancy treatment; Secondly, in Figure (a) and Figure (b), we can see that the color of the picture is more prominent; Finally, the problem of obvious seams caused by uneven brightness in the four pictures has been improved, and the objective indicators are shown in Table 1.

It can be seen from Table 1 that the PSNR and SSIM of the mosaic with color constancy module have been significantly improved, and the data have been significantly improved. The PSNR value of the four groups of data has increased by 3.555 db on average, and the SSIM has increased by 0.16 on average.

Experiment 2: Comparison of different stitching algorithms

In order to further prove the effectiveness of the splicing results, this paper has conducted detailed comparative experiments with Autostitch, SPHP [14] , AANAP [5] and literature [15] , as shown in Figure 6. It can be seen that the use of Autostitch software for splicing will lead to the inability to maintain the straight lines of some pictures, and distortion around each picture after splicing. When there are straight lines around, the straight lines cannot be maintained. For example, in figure a, the straight lines at the lower edge cannot be maintained. Compared with Autostitch software, SPHP method can obtain stitched images with better straight line retention and less obvious seam, but it still fails

Figure 5. Stitching results w/o color constancy module V.S stitching results w/o color constancy module.

Table 1. Comparison of PSNR and SSIM.

to solve the problem of brightness difference. AANAP [5] algorithm and ref [15] algorithm have obvious seams, and the brightness difference between the seams and other positions is obvious.

The objective indicators of different splicing algorithms are shown in Figure 7, the test image is the display image in Figure 6. From the data in the table, it can be seen that the method in this paper is obviously superior to other methods in terms of objective indicators in the nonferrous smelting scenario.

Figure 6. Comparison of results of different stitching methods.

Figure 7. Comparison of results of different stitching methods.

4. Conclusion

In order to solve the problems of low brightness of some scenes and small monitoring field of view caused by uneven illumination in the non-ferrous smelting scene, this paper proposes a non-ferrous smelting scene image mosaic method based on SIFT and color constancy. Compared with the traditional mosaic method, this paper introduces the preprocessing module and the color constancy module based on matching points. First, the image to be stitched is preprocessed to improve the accuracy of subsequent feature extraction and matching. Then, the preprocessed results are feature extracted and matched. After obtaining the feature matching information, the color constancy processing is performed in combination with the inter frame information between matching points, so as to effectively eliminate the brightness and color differences between the images to be stitched. Finally, multi band image fusion is performed. The experimental results show that in the non-ferrous smelting scene, the proposed method has made significant progress in the subjective and objective evaluation, and effectively eliminated the problems such as uneven brightness of panoramic image and obvious seam. However, the method proposed in this article currently only performs well in non ferrous smelting scenarios. Further improvements will be made on this basis to apply the algorithm to more scenarios.

Conflicts of Interest

The authors declare no conflicts of interest regarding the publication of this paper.

References

[1] Gao, J., Kim, S.J. and Brown, M.S. (2011) Constructing Image Panoramas Using Dual-Homography Warping. 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Colorado Springs, CO, 20-25 June 2011, 49-56.
https://doi.org/10.1109/CVPR.2011.5995433
[2] Lin, W.Y., Liu, S., Matsushita, Y., et al. (2011) Smoothly Varying Affine Stitching. 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Colorado Springs, CO, 20-25 June 2011, 345-352.
https://doi.org/10.1109/CVPR.2011.5995314
[3] Zaragoza, J., Chin, T.J., Brown, M.S., et al. (2013) As-Projective-As-Possible Image Stitching with Moving DLT. 2013 IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, 23-28 June 2013, 2339-2346.
https://doi.org/10.1109/CVPR.2013.303
[4] Chen, J., Li, N. and Liao, T. (2018) Graph-Based Hypothesis Generation for Parallax-Tolerant Image Stitching.
[5] Lin, C.C., Pankanti, S.U., Ramamurthy, K.N., et al. (2015) Adaptive As-Natural-as-Possible Image Stitching. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, 7-12 June 2015, 1155-1163.
https://doi.org/10.1109/CVPR.2015.7298719
[6] Wei, C., Wang, W., Yang, W., et al. (2018) Deep Retinex Decomposition for Low-Light Enhancement.
[7] Zhang, Y., Zhang, J. and Guo, X. (2019) Kindling the Darkness: A Practical Low-Light Image Enhancer. Proceedings of the 27th ACM International Conference on Multimedia, 1632-1640.
[8] Jobson, D.J., Rahman, Z.U. and Woodell, G.A. (1997) Properties and Performance of a Center/Surround Retinex. IEEE Transactions on Image Processing, 6, 451-462.
https://doi.org/10.1109/83.557356
[9] Gatta, C., Rizzi, A. and Marini, D. (2002) ACE: An Automatic Color Equalization Algorithm. Conference on Colour in Graphics, Imaging, and Vision, 1, Article ID: art00067.
https://doi.org/10.2352/CGIV.2002.1.1.art00067
[10] Getreuer, P. (2012) Automatic Color Enhancement (ACE) and Its Fast Implementation. Image Processing on Line, 2, 266-277.
https://doi.org/10.5201/ipol.2012.g-ace
[11] Wang, Z. and Bovik, A.C. (2002) A Universal Image Quality Index. IEEE Signal Processing Letters, 9, 81-84.
https://doi.org/10.1109/97.995823
[12] Sheikh, H.R., Sabir, M.F. and Bovik, A.C. (2006) A Statistical Evaluation of Recent Full Reference Image Quality Assessment Algorithms. IEEE Transactions on Image Processing, 15, 3440-3451.
https://doi.org/10.1109/TIP.2006.881959
[13] Wang, Z., Bovik, A.C., Sheikh, H.R., et al. (2004) Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Transactions on Image Processing, 13, 600-612.
https://doi.org/10.1109/TIP.2003.819861
[14] Jia, J.Y., et al. (2013) Tree-Based Panoramic Video Stitching. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35, 2731-2743.
[15] Zhong, M.Z., Tang, Z.T., Wang, Y.H., et al. (2022) Multi-Threshold SIFT Image Stitching Algorithm Based on Texture Classification. Computer Simulation, 39, 364-368.

Copyright © 2024 by authors and Scientific Research Publishing Inc.

Creative Commons License

This work and the related PDF file are licensed under a Creative Commons Attribution 4.0 International License.