1、June 7, 2006 13:12 WSPC/115-IJPRAI SPI-J068 00480International Journal of Pattern Recognitionand Artificial IntelligenceVol. 20, No. 4 (2006) 609628c World Scientific Publishing CompanyFLEXIBLE IMAGE BLENDING FOR IMAGE MOSAICINGWITH REDUCED ARTIFACTSWENYI ZHAOSarno Corporation, 201 Washington RoadPr
2、inceton, NJ 08540, USAwyzhaoieee.orgImage mosaicing involves geometric alignment among video frames and image composit-ing or blending. For dynamic mosaicing, image mosaics are constructed dynamicallyalong with incoming video frames. Consequently, dynamic mosaicing demands ecientoperations for both
3、alignment and blending in order to achieve real-time performance.In this paper, we focus on ecient image blending methods that create good-qualityimage mosaics from any number of overlapping frames. One of the driving forces forecient image processing is the huge market of mobile devices such as cel
4、l phones,PDAs that have image sensors and processors. In particular, we show that it is possibleto have ecient sequential implementations of blending methods that simultaneouslyinvolve all accumulated video frames. The choices of image blending include traditionalaveraging, overlapping and flexible
5、ones that take into consideration temporal order ofvideo frames and user control inputs. In addition, we show that artifacts due to mis-alignment, image intensity dierence can be significantly reduced by eciently applyingweighting functions when blending video frames. These weighting functions are b
6、asedon pixel locations in a frame, view perspective and temporal order of this frame. Oneinteresting application of flexible blending is to visualize moving objects on a mosaicedstationary background. Finally, to correct for significant exposure dierence in videoframes, we propose a pyramid extensio
7、n based on intensity matching of aligned imagesat the coarsest resolution. Our experiments with real image sequences demonstrate theadvantages of the proposed methods.Keywords: Flexible image blending; sequential implementation; mosaics with reducedartifacts; pyramid-based exposure compensation; emb
8、edded systems.1. IntroductionRecent advances on embedded systems hold great promises to make small embeddeddevices more capable and intelligent. For example, the devices become smaller andsmaller while its computational power, data storage and memory size remain thesame or even increase. With powerf
9、ul embedded processors (e.g. ARMa)andoper-ation systems (e.g. Symbianb), there seems to be endless opportunities to empowerahttp:/ J. Patt. Recogn. Artif. Intell. 2006.20:609-628. Downloaded from by DALIAN UNIVERSITY OF TECHNOLOGY on 09/14/13. For personal use only.June 7, 2006 13:12 WSPC/115-IJPRAI
10、 SPI-J068 00480610 W. ZhaoFig. 1. A typical transition from a dedicated powerful vision system (Pyramid Vision Ltd.,1990s) to a dedicated powerful vision chip (Sarno Corporation, 2000), and to more recently smallembedded devices such as cell phones and PDAs that are inexpensive and gradually becomin
11、gpowerful.increasingly popular embedded devices. Figure 1 illustrates a typical transitionfrom a dedicated vision system7to a dedicated vision chip,23and to small embed-ded devices that still lack in computational power. There already exist amazingtechniques that are built for embedded devices, for
12、example, the Okao Vision FaceRecognition Software for cell phones from Omron1and the Video Mosaicer2forcreating panoramic images for cell phones from QinetiQ. Yet, significant chal-lenges exist for transferring state-of-the-art technical capabilities developed overthe years based on special hardware
13、 and powerful PC to small embedded systemswithout sacrificing performance. In this paper, we focus on one particular tech-nique, video mosaicing that has been studied extensively5,11,14,21and continue tofind new applications.15,16The goal is to have algorithms that not only rival thebest available b
14、ut are more ecient to implement on embedded devices.Image mosaicing5,11,14,21stitches small input images to create a large mosaicimage. Two types of mosaicing exist: still mosaicing where a still mosaic imageis created by stitching together all input images,11and dynamic mosaicing wheremosaic images
15、 are created dynamically along with incoming video frames.7Twomajor steps involved in constructing image mosaics are geometric alignment andimage compositing or blending. Many techniques have been developed to handle thegeometric alignment.17,19Only possible with special hardware ten years ago,7real
16、-time image alignment is now a common practice with powerful PCs. The focus ofthis paper is to propose ecient image blending methods that can be used for bothreal-time dynamic mosaicing and ecient still mosaicing to create good-qualitymosaics from any number of overlapping images.More specifically,
17、we propose flexible blending methods where weighting func-tions for each video frame are computed individually in local frame coordinatesystems. Most state-of-the-art algorithms need to first compute all possible imageboundaries (in general, computationally expensive) and then decide the weightingfu
18、nctions based on these boundaries. As a result, they are not tailored towardsreal-time implementation. On the contrary, our methods can be implemented veryInt. J. Patt. Recogn. Artif. Intell. 2006.20:609-628. Downloaded from by DALIAN UNIVERSITY OF TECHNOLOGY on 09/14/13. For personal use only.June
19、7, 2006 13:12 WSPC/115-IJPRAI SPI-J068 00480Flexible Image Blending for Image Mosaicing with Reduced Artifacts 611eciently with results comparable to those of complex state-of-the-art algorithms.The flexible blending methods provide the following unique advantages: Capability to produce great result
20、s and potential for being implemented onembedded system for real-time performance. This is in contrast to state-of-artalgorithms where weighting functions dependent on boundaries among all framesare computed in the global mosaic coordinate system. As a result, they are com-putationally expensive and
21、 cannot be implemented sequentially. Comprehensive treatment of geometry, time and user control. The individualweighting function takes into consideration temporal order of a video frame,perspective of this video frame with respect to the global coordinate of the mosaicand user control inputs. Capab
22、ility to handle exposure imbalance among frames. To remove visible seamsdue to significant dierences in image intensity among video frames, multi-resolution flexible blending methods are proposed to explicitly compensate forexposure dierence before blending image pixels.In Fig. 2, we illustrate the
23、impact of image blending upon the quality of imagemosaics using a pair of stereo images by averaging and applying individual weightingfunctions. We notice that in Fig. 2(b) not only boundary seams are removed, butartifacts (ghosting) due to local alignment error/geometric distortion are visuallyremo
24、ved. This is made possible by applying continuous but fast decaying weightingfunctions that act more like selection, for example, (0.53x+0.13y)/(0.53+0.13) xwhere x and y are pixel values from two video frames.The remainder of this paper is organized as follows: after reviewing relatedwork in Sec. 2
25、, we present the flexible blending methods in Sec. 3. We then presentexperimental results in Sec. 4. In Sec. 5, we discuss the extension of flexible blendingmethods to multiresolution flexible blending methods that handle the exposureissue. Finally, we conclude our paper in Sec. 6.2. Related WorkThe
26、 task of image blending is to determine the value of a mosaic pixel based onpixels from all warped images. Many factors aect the quality of image mosaics,including image alignment, lens distortion, pixel intensity dierence among frames,and perspectives of each frame. There is a long history of study
27、 on how to creategood quality mosaics. The majority of image blending methods create a mosaicpixel based on a weighted sum of warped frame pixels. We refer the pixel-wiseweights per frame as the weighting functions.The simplest weighting function is a flat function that weighs all pixels equally.Ima
28、ge blending using the flat weighting function leads to visible edges or seamsalong image boundaries Fig. 2(a). To reduce seams, a linear ramp weighting func-tion can be used across the boundary region of two adjacent images.11A similartechnique (feathering algorithm) was proposed in Ref. 20 to reduc
29、e seams. ImageInt. J. Patt. Recogn. Artif. Intell. 2006.20:609-628. Downloaded from by DALIAN UNIVERSITY OF TECHNOLOGY on 09/14/13. For personal use only.June 7, 2006 13:12 WSPC/115-IJPRAI SPI-J068 00480612 W. Zhao(a)(b)(c)Fig. 2. Image mosaics constructed from two stereo images based on (a) a flat
30、weighting function(averaging), (b) a bi-cubic decaying weighting function centered in the optical center of a frame.The two stereo images are plotted in (c). White arrows indicate places where significant dierenceis visible. For detailed explanations, please see Fig 3.Int. J. Patt. Recogn. Artif. In
31、tell. 2006.20:609-628. Downloaded from by DALIAN UNIVERSITY OF TECHNOLOGY on 09/14/13. For personal use only.June 7, 2006 13:12 WSPC/115-IJPRAI SPI-J068 00480Flexible Image Blending for Image Mosaicing with Reduced Artifacts 613feathering assigns a pixels weight based on its distance to the boundary
32、. A bet-ter approach that intends to compute the best possible weighting function acrossboundaries was proposed in Ref. 14. However, it involves an expensive iterativerelaxation procedure.Recognizing the impact of the width of a transition zone centered across animage boundary, authors in Ref. 3 pro
33、posed a multiresolution method based onLaplacian pyramid.4The basic motivation of this approach is that the width of thetransition zone should be comparable to the size of features in images. To handlefeatures of dierent sizes, a fusion approach3was taken that first decomposes imagesinto dierent res
34、olutions, creating component mosaics at each resolution using tran-sition zones of dierent widths; then blends all component mosaics to produce thefinal mosaic. In Ref. 8, DWT (discrete wavelet transform) rather than Laplacianpyramid is applied to create multiresolution mosaic. An alternative approa
35、ch tomultiresolution image blending is to perform operations in the gradient domain.10These gradient-domain methods construct the mosaic image by an optimizationprocess that compares the gradients of the mosaic image and all video frames.Improved results were shown compared to multiresolution blendi
36、ng and featheringalgorithms.Other than image blending, many methods have focused on geometry in order toremove seams and ghosting. For example, local alignment (block-based optical flow)is applied to local overlapping regions after global alignment to remove ghostingcaused by misalignment.19Another
37、interesting method to remove seams geomet-rically is the use of image strips that are perpendicular to the image flow.12,13The mosaic manifold is defined by the motion of a capturing camera. By perfectlyaligning 1D vertical strips along the seam, this method works very well with a 1Dscanning paths b
38、ut not with a 2D scanning path.Other methods consider handling dynamic geometry, for example, movingobjects within video frames. In Ref. 6, a method was proposed to segment themosaic into disjoint regions and sample pixels in each mosaic region from a sin-gle input image. To avoid blurred areas due
39、to moving objects, each movingobject is assumed to be fully contained in at least one input image. To handlethe cases of multiple overlapping regions that contain moving objects, a vertexcover algorithm was proposed to selectively remove all but one instance of eachobject.22One common issue with the
40、se methods, blending ones or geometry ones, is thatthey are not tailored towards real-time, especially dynamic mosaicing systems. Theyfirst need to compute all possible image boundaries (which could be very compli-cated), and then decide the weighting functions based on these boundaries. In short,al
41、l the operations are carried out in the mosaic coordinate system. On the contrary,the flexible blending methods proposed in this paper start from local frame coor-dinate systems and end up in the mosaic coordinate system through simple imagewarping, regardless of how arbitrary the boundaries are in
42、the mosaic coordinatesystem. In addition to removing seams, the proposed blending methods carry outInt. J. Patt. Recogn. Artif. Intell. 2006.20:609-628. Downloaded from by DALIAN UNIVERSITY OF TECHNOLOGY on 09/14/13. For personal use only.June 7, 2006 13:12 WSPC/115-IJPRAI SPI-J068 00480614 W. Zhaod
43、eghosting quite eectively. The proposed methods also consider perspective andtemporal order of video frames and user input controls.Finally, there exist methods that handle exposure dierence explicitly. InRef. 22, an iterated method based on image blocks (e.g. 32 32) was proposed tosolve the transfe
44、r functions from corresponding image blocks to the mosaic block.An additional step is then applied to guarantee the smoothness of the final imagecomposite. In this paper, a dierent method based on pyramid decomposition andreconstruction is proposed to explicitly correct for exposure imbalance. As a
45、result,not only seams are removed but exposure-consistent mosaics are created. Otherrelated work actually explores exposure dierence to create images with extendeddynamic range.9,183. Flexible Image BlendingBefore we introduce flexible image blending, we would like to demonstrate theimpact of image
46、blending upon the quality of image mosaicing. We take the mosaic-ing result in Fig. 2 as an example and plot regions where image blending has madea significant impact. It is obvious from Fig. 3 that mosaicing artifacts have beensignificantly reduced: (i) artifacts (ghosting) due to alignment error F
47、igs. 3(a) and3(b), (ii) artifacts due to geometric distortion (and alignment) Fig. 3(c), (iii) arti-facts due to intensity dierence Fig. 3(d). From these comparisons we can see thatthe proposed blending functions can act more like selecting rather than averagingwhen the distances from a mosaic pixel
48、 to respective warped image optical centersare rather dierent.(a) (b) (c) (d)Fig. 3. Close-up comparison of image mosaics constructed in Fig. 2. Four regions are plotted here:(a) misaligned car, (b) misaligned pole, (c) distorted tree, and (d) road with dierent intensities.For each region, there are
49、 four images arranged in the same order: the top-left and top-right arefrom warped versions of the original images Fig. 2(c); the bottom-left is the result of averagingthe warped images based on the flat weighting function Fig. 2(a) and the bottom-right is basedon individual weighting functions Fig. 2(b).Int. J. Patt. Recogn. Artif. Intell. 2006.20:609-628. Downloaded from www.worldsci
