1、Image Features,CSE 576, Spring 2005,3/31/2005,CSE 576: Computer Vision,2,About me,Ph. D., Carnegie Mellon, 1988 Researcher, Cambridge Research Lab at DEC, 1990-1995 Senior Researcher, Interactive Visual Media Group, Microsoft, 1995- Research interests: computer vision (stereo, motion), computer grap
2、hics (image-based rendering), data-parallel programming,3/31/2005,CSE 576: Computer Vision,3,Todays lecture,What is computer vision? Scale-space and pyramids What are good features? Feature detection Feature descriptors (Next lecture: feature matching)Project 1description and demo Ian Simon,What is
3、Computer Vision?,3/31/2005,CSE 576: Computer Vision,5,What is Computer Vision?,Image Understanding (AI, behavior) A sensor modality for robotics Computer emulation of human vision Inverse of Computer Graphics,3/31/2005,CSE 576: Computer Vision,6,Intersection of Vision and Graphics,3/31/2005,CSE 576:
4、 Computer Vision,7,Computer Vision Trucco&Verri98,3/31/2005,CSE 576: Computer Vision,8,Image-Based Modeling,3/31/2005,CSE 576: Computer Vision,9,Related disciplines,Image Processing Scientific / medical imaging Pattern Recognition Computer Graphics Learning Artificial Intelligence Visual Neuroscienc
5、e Applied Mathematics,3/31/2005,CSE 576: Computer Vision,10,Mathematics,What kinds of mathematics are used? Signal and image processing Euclidean and projective geometry Vector calculus Partial differential equations Optimization Probabilistic estimation,3/31/2005,CSE 576: Computer Vision,11,Applica
6、tions,Geometric reconstruction: modeling, forensics, special effects (ILM, RealVis,2D3) Image and video editing (Avid, Adobe) Webcasting and Indexing Digital Video (Virage) Scientific / medical applications (GE),3/31/2005,CSE 576: Computer Vision,12,Applications,Tracking and surveillance (Sarnoff) F
7、ingerprint recognition (Digital Persona) Biometrics / iris scans (Iridian Technologies) Vehicle safety (MobilEye) Drowning people (VisionIQ Inc) Optical motion capture (Vicon),3/31/2005,CSE 576: Computer Vision,13,Image Morphing,3/31/2005,CSE 576: Computer Vision,14,Panoramic Mosaics,+ + + =,3/31/20
8、05,CSE 576: Computer Vision,15,3D Shape Reconstruction,Debevec, Taylor, and Malik, SIGGRAPH 1996,3/31/2005,CSE 576: Computer Vision,16,Face Modeling,3/31/2005,CSE 576: Computer Vision,17,View Morphing,Morph between pair of images using epipolar geometry Seitz & Dyer, SIGGRAPH96,3/31/2005,CSE 576: Co
9、mputer Vision,18,Virtualized RealityTM,Takeo Kanade, CMU collect video from 50+ stream reconstruct 3D model sequences http:/www.cs.cmu.edu/afs/cs/project/VirtualizedR/www/VirtualizedR.html,3/31/2005,CSE 576: Computer Vision,19,Virtualized RealityTM,Takeo Kanade, CMU generate new video steerable vers
10、ion used for SuperBowl XXV “eye vision” system,3/31/2005,CSE 576: Computer Vision,20,Edge detection and editing,Elder, J. H. and R. M. Goldberg. “Image Editing in the Contour Domain,“ Proc. IEEE: Computer Vision and Pattern Recognition, pp. 374-381, June, 1998.,3/31/2005,CSE 576: Computer Vision,21,
11、Image Enhancement,High dynamic range photography Debevec et al.97; Mitsunaga & Nayar99 combine several different exposures together,3/31/2005,CSE 576: Computer Vision,22,Projects,Lets look at what students have done in previous years Stanford http:/www.stanford.edu/class/cs223b/winter01-02/projects.
12、html CMU http:/www-2.cs.cmu.edu/ph/869/www/869.html (available) UW http:/www.cs.washington.edu/education/courses/cse590ss/01wi/ GA Tech http:/www.cc.gatech.edu/classes/AY2002/cs4480_spring/,3/31/2005,CSE 576: Computer Vision,23,Todays lecture,What is computer vision? Scale-space and pyramids What ar
13、e good features? Feature detection Feature descriptors (Next lecture: feature matching)Project 1description and demo Ian Simon,Image Pyramids,3/31/2005,CSE 576: Computer Vision,25,Image Pyramids,3/31/2005,CSE 576: Computer Vision,26,Pyramid Creation,“Laplacian” Pyramid Created from Gaussian pyramid
14、by subtraction Ll = Gl expand(Gl+1),filter mask,“Gaussian” Pyramid,3/31/2005,CSE 576: Computer Vision,27,Octaves in the Spatial Domain,Bandpass Images,Lowpass Images,3/31/2005,CSE 576: Computer Vision,28,Pyramids,Advantages of pyramids Faster than Fourier transform Avoids “ringing” artifacts Many ap
15、plications small images faster to process good for multiresolution processing compression progressive transmission(传送) Known as “MIP-maps” in graphics community Precursor(ancestor) to wavelets Wavelets also have these advantages,3/31/2005,CSE 576: Computer Vision,29,Laplacian level 4,Laplacian level
16、 2,Laplacian level 0,left pyramid,right pyramid,blended pyramid,3/31/2005,CSE 576: Computer Vision,30,Pyramid Blending,3/31/2005,CSE 576: Computer Vision,31,smoothed original (scaled by 4, offset +128),original,smoothed (5x5 Gaussian),why does this work?,3/31/2005,CSE 576: Computer Vision,32,Scale s
17、pace (Witkin 83),larger,Gaussian filtered signal,Zero crossings,3/31/2005,CSE 576: Computer Vision,33,Scale space: insights,As the scale is increased edge position can change edges can disappear new edges are not created,Bottom linekey need to consider edges at different scales (or else know what sc
18、ale you care about),3/31/2005,CSE 576: Computer Vision,34,Todays lecture,What is computer vision? Scale-space and pyramids What are good features? Feature detection Feature descriptors (Next lecture: feature matching)Project 1description and demo Ian Simon,These slides adapted from: Matching with In
19、variant Features,Darya Frolova, Denis Simakov The Weizmann Institute of Science March 2004,and Real-time Object Recognition using Invariant Local Image Features,David Lowe Computer Science Department University of British Columbia NIPS 2003 Tutorial,3/31/2005,CSE 576: Computer Vision,37,Invariant Lo
20、cal Features,Image content is transformed into local feature coordinates that are invariant to translation, rotation, scale, and other imaging parameters,SIFT Features,3/31/2005,CSE 576: Computer Vision,38,Advantages of local features,Locality: features are local, so robust to occlusion and clutterc
21、haos (no prior segmentation) Distinctiveness: individual features can be matched to a large database of objects Quantity: many features can be generated for even small objects Efficiency: close to real-time performance Extensibility: can easily be extended to wide range of differing feature types, w
22、ith each adding robustness,3/31/2005,CSE 576: Computer Vision,39,More motivation,Feature points are used also for: Image alignment (homography, fundamental matrix) 3D reconstruction Motion tracking Object recognition Indexing and database retrieval Robot navigation other,3/31/2005,CSE 576: Computer
23、Vision,40,Harris corner detector,C.Harris, M.Stephens. “A Combined Corner and Edge Detector”. 1988,3/31/2005,CSE 576: Computer Vision,41,The Basic Idea,We should easily recognize the point by looking through a small window Shifting a window in any direction should give a large change in intensity,3/
24、31/2005,CSE 576: Computer Vision,42,Harris Detector: Basic Idea,“flat” region: no change in all directions,“edge”: no change along the edge direction,“corner”: significant change in all directions,3/31/2005,CSE 576: Computer Vision,43,Harris Detector: Mathematics,Change of intensity for the shift u,
25、v:,3/31/2005,CSE 576: Computer Vision,44,Harris Detector: Mathematics,For small shifts u,v we have a bilinear approximation:,where M is a 22 matrix computed from image derivatives:,3/31/2005,CSE 576: Computer Vision,45,Harris Detector: Mathematics,Intensity change in shifting window: eigenvalue anal
26、ysis,1, 2 eigenvalues of M,direction of the slowest change,direction of the fastest change,(max)-1/2,(min)-1/2,Ellipse E(u,v) = const,3/31/2005,CSE 576: Computer Vision,46,Harris Detector: Mathematics,1,2,“Corner” 1 and 2 are large, 1 2; E increases in all directions,1 and 2 are small; E is almost c
27、onstant in all directions,“Edge” 1 2,“Edge” 2 1,“Flat” region,Classification of image points using eigenvalues of M:,3/31/2005,CSE 576: Computer Vision,47,Harris Detector: Mathematics,Measure of corner response:,(k empirical经验的 constant, k = 0.04-0.06),3/31/2005,CSE 576: Computer Vision,48,Harris De
28、tector: Mathematics,1,2,“Corner”,“Edge”,“Edge”,“Flat”,R depends only on eigenvalues of MR is large for a cornerR is negative with large magnitude for an edge|R| is small for a flat region,R 0,R 0,R 0,|R| small,3/31/2005,CSE 576: Computer Vision,49,Harris Detector,The Algorithm: Find points with larg
29、e corner response function R (R threshold) Take the points of local maxima of R,3/31/2005,CSE 576: Computer Vision,50,Harris Detector: Workflow,3/31/2005,CSE 576: Computer Vision,51,Harris Detector: Workflow,Compute corner response R,3/31/2005,CSE 576: Computer Vision,52,Harris Detector: Workflow,Fi
30、nd points with large corner response: Rthreshold,3/31/2005,CSE 576: Computer Vision,53,Harris Detector: Workflow,Take only the points of local maxima of R,3/31/2005,CSE 576: Computer Vision,54,Harris Detector: Workflow,3/31/2005,CSE 576: Computer Vision,55,Harris Detector: Summary,Average intensity
31、change in direction u,v can be expressed as a bilinear form: Describe a point in terms of eigenvalues of M: measure of corner response A good (corner) point should have a large intensity change in all directions, i.e. R should be large positive,3/31/2005,CSE 576: Computer Vision,56,Harris Detector:
32、Some Properties,Rotation invariance,Ellipse rotates but its shape (i.e. eigenvalues) remains the same,Corner response R is invariant to image rotation,3/31/2005,CSE 576: Computer Vision,57,Harris Detector: Some Properties,Partial invariance to affine intensity change,Only derivatives are used = inva
33、riance to intensity shift I I + b,3/31/2005,CSE 576: Computer Vision,58,Harris Detector: Some Properties,But: non-invariant to image scale!,All points will be classified as edges,Corner !,3/31/2005,CSE 576: Computer Vision,59,Harris Detector: Some Properties,Quality of Harris detector for different
34、scale changes,Repeatability rate:,# correspondences # possible correspondences,C.Schmid et.al. “Evaluation of Interest Point Detectors”. IJCV 2000,3/31/2005,CSE 576: Computer Vision,60,Models of Image Change,Geometry Rotation Similarity (rotation + uniform scale) Affine (scale dependent on direction
35、) valid for: orthographic camera, locally planar object Photometry光度 Affine intensity change (I a I + b),3/31/2005,CSE 576: Computer Vision,61,Rotation Invariant Detection,Harris Corner Detector,C.Schmid et.al. “Evaluation of Interest Point Detectors”. IJCV 2000,3/31/2005,CSE 576: Computer Vision,62
36、,Scale Invariant Detection,Consider regions (e.g. circles) of different sizes around a point Regions of corresponding sizes will look the same in both images,3/31/2005,CSE 576: Computer Vision,63,Scale Invariant Detection,The problem: how do we choose corresponding circles independently in each imag
37、e?,3/31/2005,CSE 576: Computer Vision,64,Scale invariance,Requires a method to repeatably select points in location and scale: The only reasonable scale-space kernel is a Gaussian (Koenderink, 1984; Lindeberg, 1994) An efficient choice is to detect peaks in the difference of Gaussian pyramid (Burt C
38、rowley & Parker, 1984 but examining more scales) Difference-of-Gaussian with constant ratio of scales is a close approximation to Lindebergs scale-normalized Laplacian (can be shown from the heat diffusion扩散 equation),3/31/2005,CSE 576: Computer Vision,65,Scale Invariant Detection,Solution: Design a
39、 function on the region (circle), which is “scale invariant” (the same for corresponding regions, even if they are at different scales),Example: average intensity. For corresponding regions (even of different sizes) it will be the same.,For a point in one image, we can consider it as a function of r
40、egion size (circle radius),3/31/2005,CSE 576: Computer Vision,66,Scale Invariant Detection,Common approach:,Take a local maximum of this function,Observation: region size, for which the maximum is achieved, should be invariant to image scale.,Important: this scale invariant region size is found in e
41、ach image independently!,3/31/2005,CSE 576: Computer Vision,67,Scale Invariant Detection,A “good” function for scale detection: has one stable sharp peak,For usual images: a good function would be a one which responds to contrast (sharp local intensity change),3/31/2005,CSE 576: Computer Vision,68,S
42、cale Invariant Detection,Functions for determining scale,Kernels:,where Gaussian,Note: both kernels are invariant to scale and rotation,(Laplacian),(Difference of Gaussians),3/31/2005,CSE 576: Computer Vision,69,Scale space: one octave倍频程 at a time,3/31/2005,CSE 576: Computer Vision,70,Key point loc
43、alization,Detect maxima and minima of difference-of-Gaussian in scale space Fit a quadratic二次 to surrounding values for sub-pixel and sub-scale interpolation (Brown & Lowe, 2002) Taylor expansion around point:Offset of extremum (use finite differences for derivatives):,3/31/2005,CSE 576: Computer Vi
44、sion,71,Sampling frequency for scale,More points are found as sampling frequency increases, but accuracy of matching decreases after 3 scales/octave,3/31/2005,CSE 576: Computer Vision,72,Eliminating unstable keypoints,Discard points with DOG value below threshold (low contrast) However, points along
45、 edges may have high contrast in one direction but low in another Compute principal curvatures from eigenvalues of 2x2 Hessian matrix, and limit ratio (Harris approach):,3/31/2005,CSE 576: Computer Vision,73,Scale Invariant Detectors,Harris-Laplacian1 Find local maximum of: Harris corner detector in
46、 space (image coordinates) Laplacian in scale,1 K.Mikolajczyk, C.Schmid. “Indexing Based on Scale Invariant Interest Points”. ICCV 2001 2 D.Lowe. “Distinctive Image Features from Scale-Invariant Keypoints”. Accepted to IJCV 2004,3/31/2005,CSE 576: Computer Vision,74,Scale Invariant Detectors,K.Mikol
47、ajczyk, C.Schmid. “Indexing Based on Scale Invariant Interest Points”. ICCV 2001,Experimental evaluation of detectors w.r.t. scale change,Repeatability rate:,# correspondences # possible correspondences,3/31/2005,CSE 576: Computer Vision,75,3/31/2005,CSE 576: Computer Vision,76,3/31/2005,CSE 576: Co
48、mputer Vision,77,Scale Invariant Detection: Summary,Given: two images of the same scene with a large scale difference between them Goal: find the same interest points independently in each image Solution: search for maxima of suitable functions in scale and in space (over the image),Methods: Harris-
49、Laplacian Mikolajczyk, Schmid: maximize Laplacian over scale, Harris measure of corner response over the image SIFT Lowe: maximize Difference of Gaussians over scale and space,3/31/2005,CSE 576: Computer Vision,78,Affine Invariant Detection,Above we considered: Similarity transform (rotation + uniform scale),Now we go on to: Affine transform (rotation + non-uniform scale),3/31/2005,CSE 576: Computer Vision,
