Report

The University of CS 4487/9587 Algorithms for Image Analysis Image Processing Basics Lecture 3 Lena Gorelick, substituting for Yuri Boykov Acknowledgements: slides from Steven Seitz, Aleosha Efros, David Forsyth, and Gonzalez & Woods Ontario The University of CS 4487/9587 Algorithms for Image Analysis Image Processing Basics Ontario Image Processing • Domain Transformation – Resize, Rotate etc. • Range Transformation – Point Processing • • • gamma correction window-center correction histogram equalization Pixel location g ( x , y ) f ( t x ( x , y ), t y ( x , y )) g ( x, y ) f ( x, N y ) Pixel Intensity g ( x , y ) t x , y ( f ( x , y )) g ( x, y ) f ( x, y ) / 2 Window = 800 Center = 1160 The University of CS 4487/9587 Algorithms for Image Analysis Image Processing Basics Ontario Image Processing • Domain Transformation – Resize, Rotate etc. • Range Transformation – Point Processing • • • gamma correction window-center correction histogram equalization g ( x , y ) f ( t x ( x , y ), t y ( x , y )) g ( x , y ) f ( 2 x ,2 y ) g ( x , y ) t x , y ( f ( x , y )) g ( x, y ) f ( x, y ) / 2 – Neighborhood Processing (Filtering) The University of Neighborhood Processing (or filtering) What is wrong with this image? Courtesy of Carlo Tomasi Courtesy of Neel Joshi How can we remove noise? • Look spatially around each pixel - neighborhood Can we use Point Processing? Readings: Forsyth & Ponce, chapters 8.1-8.2 Ontario The University of Neighborhood Processing (or filtering) Lets reshuffle all pixels within the image Has Spatial Information Looks noisy Point Processing will have the same effect • Gets intensity of a single pixel as an input • Unaware of spatial information Neighborhood Processing • Takes “spatial information” into account • Images contain “spatial information” Readings: Forsyth & Ponce, chapters 8.1-8.2 Ontario The University of Neighborhood Processing (filtering) Linear image transforms Assume 1D function f [i ] Ontario 0 1 0 0 M 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 1 g M f ? g [i ] M [i , j ] f [ j ] j 0 0 0 0 0 0 0 0 The University of Neighborhood Processing (filtering) Linear image transforms Ontario 2 1 0 0 f [i ] M 1 / 2 0 0 0 0 Assume 1D function 0 0 0 0 0 0 1 0 0 0 0 0 1 1 0 0 0 0 0 0 1 0 1 1 0 1 0 0 0 0 0 0 0 1 1 0 0 0 0 0 1 1 0 0 0 0 0 1 g M f ? g [i ] M [i , j ] f [ j ] j 0 0 0 0 0 0 0 1 The University of Neighborhood Processing (filtering) Linear shift-invariant filters Ontario matrix M A commonly used pattern Represented by a kernel (or a mask) h [ a b c ] * a 0 0 M 0 0 0 0 * 0 0 0 0 0 b c 0 0 0 0 a b c 0 0 0 0 0 a 0 b a c b 0 c 0 0 0 0 0 a b c 0 0 0 0 a b 0 0 0 0 0 * 0 0 0 0 0 0 0 * (i - 1) i (i + 1) For 2k+1 size kernel g M f k g [i ] uk h [u ] f [i u ] The University of Neighborhood Processing (filtering) 2D filtering Ontario A 2D function f [ i , j ] Can be filtered by a 2D filter (kernel) To get a new image k g [i , j ] h[ u , v ] k h [u , v ] f [i u , j v ] uk vk This is called a cross-correlation operation Or a sliding dot-product g h f The University of Neighborhood Processing (filtering) 2D filtering Ontario Cross-correlation in which the filter is flipped horizontally and vertically is called convolution g h* f k k g [i , j ] h[ u , v ] f [ i u , j v ] uk vk k k uk vk h [u , v ] f [i u , j v ] The University of Neighborhood Processing (filtering) Convolution vs. Cross-Correlation If the kernel is symmetric h (u , v ) h ( u , v ) convolution = cross-correlation Ontario Neighborhood Processing 2D filtering Noise Types of noise: • Salt and pepper noise • Impulse noise • Gaussian noise Due to • • • • transmission errors dead CCD pixels specks on lens can be specific to a sensor The University of Ontario The University of Neighborhood Processing Practical Noise Reduction How can we remove noise? Replace each pixel with the average of a kxk window around it Ontario 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100 130 110 120 110 0 0 0 0 0 110 90 100 90 100 0 0 0 0 0 130 100 104 90 130 110 0 0 0 0 0 120 100 130 110 120 0 0 0 0 0 90 110 80 120 100 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 The University of Neighborhood Processing (filtering) Mean filtering Ontario 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 0 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 f [ x, y ] g[ x, y ] The University of Neighborhood Processing (filtering) Mean filtering Ontario 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 10 20 30 30 30 20 10 0 0 0 90 90 90 90 90 0 0 0 20 40 60 60 60 40 20 0 0 0 90 90 90 90 90 0 0 0 30 60 90 90 90 60 30 0 0 0 90 90 90 90 90 0 0 0 30 50 80 80 90 60 30 0 0 0 90 0 90 90 90 0 0 0 30 50 80 80 90 60 30 0 0 0 90 90 90 90 90 0 0 0 20 30 50 50 60 40 20 0 0 0 0 0 0 0 0 0 0 10 20 30 30 30 30 20 10 0 0 90 0 0 0 0 0 0 0 10 10 10 0 0 0 0 0 0 0 0 0 0 0 f [ x, y ] 0 g[ x, y ] 0 0 0 The University of Neighborhood Processing (filtering) Effect of mean filters Gaussian noise Salt & Pepper noise 3x3 Side effect - blur Ontario 5x5 7x7 The University of Neighborhood Processing (filtering) Mean kernel Ontario What’s the kernel for a 3x3 mean filter? 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 0 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 f [ x, y ] h[ u , v ] The University of Neighborhood Processing (filtering) Mean kernel Ontario What’s the kernel for a 3x3 mean filter? 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 1/9 1/9 1/9 0 0 0 90 90 90 90 90 0 0 1/9 1/9 1/9 0 0 0 90 0 90 90 90 0 0 1/9 1/9 1/9 0 0 0 90 90 90 90 90 0 0 h[ u , v ] 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 f [ x, y ] The University of Neighborhood Processing (filtering) Mean kernel Ontario What’s the kernel for a 3x3 mean filter? 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 0 90 90 90 0 0 1 1 1 0 0 0 90 90 90 90 90 0 0 h[ u , v ] 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 f [ x, y ] 1 1 1 1/9 1 1 1 Equal weight to all pixels within the neighborhood The University of Neighborhood Processing (filtering) Gaussian Filtering Ontario A Gaussian kernel gives less weight to pixels 0 0 0 0 0 0 0 0 0 0 further from the center 0 0 0 0 0 0 0 0 0 0 of the window 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 0 90 90 90 0 0 1 2 1 0 0 0 90 90 90 90 90 0 0 h[ u , v ] 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0 is a discrete approximation of a Gaussian function: 0 0 0 0 0 0 0 0 0 0 f [ x, y ] 1 2 1 1 16 2 4 2 The University of Neighborhood Processing (filtering) Mean vs. Gaussian filtering Ontario Input image Mean Filtering Mean filter, 20x20 Gaussian filtering Gaussian filter, 20x20, σ = 5 The University of Neighborhood Processing (filtering) Mean vs. Gaussian filtering Ontario The University of Neighborhood Processing (filtering) Gaussian Filtering Ontario Low-pass filter Smooth color variation (low frequency) is preserved Sharp edges (high frequency) are removed The University of Neighborhood Processing (filtering) Median filters Ontario A Median Filter operates over a window by selecting the median intensity in the window. Image credit: Wikipedia – page on Median Filter The University of Neighborhood Processing (filtering) Median filters Ontario Is a median filter a kind of convolution? No, median filter is an example of non-linear filtering Median ( f 1 ( x ) f 2 ( x )) Median ( f 1 ( x )) Median ( f 2 ( x )) 1 1 Median ( 1 1 1 1 Median ( 0 1 0 3 1 1 0 1 1 1 ) Median ( 0 1 1 1 0 0 ) Median ( 1 0 1 0 1 0 1 2 1 1 0 0 1 1 0 1 2 1 0 1 ) 0 0 1 ) 11 2 0 The University of Salt&Pepper Noise Ontario The University of Gaussian noise Ontario The University of Neighborhood Processing (filtering) Median filters What advantage does a median filter have over a mean filter? Better at removing Salt & Pepper noise Disadvantage: Slow Ontario The University of Neighborhood Processing (filtering) Derivatives and Convolution Image Derivatives and Gradients • Used for Edge/Corner Detection • Computed with Finite Differences Filters Laplacian of Gaussians (LoG) Filter • Used for Edge/Blob Detection and Image Enhancement • Approximated using Difference of Gaussians Ontario The University of Reading: Forsyth & Ponce, 8.1-8.2 First Derivative Ontario Recall Sharp changes in gray level of the input image correspond to “peaks or valleys” of the first-derivative of the input signal. http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf The University of Reading: Forsyth & Ponce, 8.1-8.2 First Derivative and convolution x f ( x h, y ) f ( x, y ) f lim h 0 h How can we approximate it for a discrete function? Is this operation shift-invariant? Is it linear? ? ? ? ? ? ? ? ? ? x [u , v ] Ontario The University of Reading: Forsyth & Ponce, 8.1-8.2 First Derivative and convolution Finite Difference Ontario f ( x a) f ( x b) x h[ f ] h f ( x h) f ( x) h h f ' ( x ) 0 x-h x-h/2 x+h x x x+h/2 Slide credit: Wikipedia The University of Reading: Forsyth & Ponce, 8.1-8.2 First Derivative and convolution Ontario Finite Difference – The order of an error can be derived using Taylor Theorem Slide credit: Wikipedia The University of Reading: Forsyth & Ponce, 8.1-8.2 First Derivative and convolution Ontario Pixel Size x h Using Finite Central Difference x f f ( x x, y ) f ( x x, y ) We need a kernel x 2 x x such that 0 0 0 1 f x f 2x 1 0 -1 0 0 0 x [u , v ] The University of Neighborhood Processing (filtering) Finite differences – responds to edges Dark = negative White = positive Gray = 0 x f Ontario The University of Neighborhood Processing (filtering) Finite differences - responding to noise x f x f Increasing noise -> (this is zero mean additive gaussian noise) Ontario x f The University of Neighborhood Processing (filtering) Finite differences and noise Ontario Finite difference filters respond strongly to noise • Noisy pixels look very different from their neighbours • The larger the noise the stronger is the response How can we eliminate the response to noise? • Most pixels in images look similar to their neighbours (even at an edge) • Smooth the image (mean/gaussian filtering) The University of Neighborhood Processing (filtering) Smoothing and Differentiation Smoothing before differentiation = two convolutions x (H f ) Convolution is associative x ( H f ) ( x H ) f ( x H ) ( y H ) Ontario The University of Neighborhood Processing (filtering) Smoothing and Differentiation Ontario ( x H ) f 1 pixel 3 pixels 7 pixels The scale of the smoothing filter affects derivative estimates, and also the semantics of the edges recovered. The University of Neighborhood Processing (filtering) Sobel kernels Ontario Yet another approximation frequently used x f 1 0 -1 1 8 x y 2 0 -2 1 8 y f 1 2 1 0 0 0 1 0 -1 -1 -2 -1 x [u , v ] y [u , v ] The University of Neighborhood Processing (filtering) Image Gradients Ontario Recall for a function of two variables f ( x , y ) The gradient at a point (x,y) fx x f f y y Gradient Magnitude || f || f f f f x y f y f ( x ) ( y ) 2 2 ( x f ) ( y f ) 2 x 2 Gradient Orientation • direction of the “steepest ascend” • orthogonal to object boundaries in the image small image gradients in low textured areas The University of Neighborhood Processing (filtering) Image Gradient for Edge Detection Ontario Typical application where image gradients are used is image edge detection • find points with large image gradients Canny edge detector suppresses non-extrema Gradient points The University of Reading: Forsyth & Ponce, 8.1-8.2 Second derivatives and convolution Ontario Peaks or valleys of the first-derivative of the input signal, correspond to “zero-crossings” of the second-derivative of the input signal. http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf The University of Neighborhood Processing (filtering) Second derivatives and convolution Ontario http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf The University of Neighborhood Processing (filtering) Second derivatives and convolution Ontario Better localized edges But more sensitive to noise http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf The University of Neighborhood Processing (filtering) Second derivatives and convolution Ontario http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf The University of Neighborhood Processing (filtering) Second Image Derivatives f f f 2 Laplace operator f 2 f x 2 f 2 y 2 Ontario [ x x ] f f y y “divergence of gradient” rotationally invariant second derivative for 2D functions 0 0 0 -1 2 -1 0 -1 0 0 2 0 0 0 0 0 -1 0 Finite Difference Second Order Derivative in x Finite Difference Second Order Derivative in y 0 -1 0 -1 4 -1 0 -1 0 The University of Neighborhood Processing (filtering) Second Image Derivatives Ontario f 2 f 2 Laplacian Zero Crossing f Used for edge detection (alternative to computing Gradient extrema) x 2 y 2 image f intensity magnitude of || image gradient http://homepages.inf.ed.ac.uk/rbf/HIPR2/zeros.htm Laplacian (2nd derivative) of the image f || f The University of Neighborhood Processing (filtering) Laplacian Filtering Ontario -Zero on uniform regions -Positive on one side of an edge -Negative on the other side -Zero at some point in between on the edge itself band-pass filter (Suppresses both high and low frequencies) http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf The University of Neighborhood Processing (filtering) Laplacian of a Gaussian (LoG) Smooth before differentiation (remember associative property of convolution) LoG ( x , y ) 1 x y 1 e 2 2 2 4 2 2 x y 2 Ontario G LoG 2 2 http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf The University of Neighborhood Processing (filtering) Laplacian of a Gaussian (LoG) Ontario Suppresses both high and low frequencies http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf The University of Neighborhood Processing (filtering) Laplacian of a Gaussian (LoG) Ontario http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf The University of Neighborhood Processing (filtering) Laplacian of a Gaussian (LoG) Ontario http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf The University of Neighborhood Processing (filtering) Laplacian of a Gaussian (LoG) Ontario Can be approximated by a difference of two Gaussians (DoG) http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf The University of Neighborhood Processing (filtering) DoG vs. LoG Ontario Separable (product decomposition) more efficient Can explain band-pass filter since Gaussian is a lowpass filter. http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf The University of Neighborhood Processing (filtering) LoG for Blob Detection Ontario Cross correlation with a filter can be viewed as comparing a little “picture” of what you want to find against all local regions in the image. Scale of blob (size ; radius in pixels) is determined by the sigma parameter of the LoG filter. http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf The University of Neighborhood Processing (filtering) LoG for Blob Detection Ontario http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf The University of Reading: Forsyth & Ponce ch.7.5 Filters and Templates Ontario Recall: filtering = cross-correlation = dot-product f(x,y) (x,y) h* f g(x,y) (x,y) Correlation Filter Mask Input Image f w f w ( x, y ) arccos h fw || h |||| f w || Output Image g ( x , y ) cos( ) Measures the angle between two vectors: the filter and the image window Measures visual similarity The University of Neighborhood Processing (filtering) Template Matching Ontario Since we divide by the norm Normalized cross-correlation f w f w ( x, y ) arccos h fw || h |||| f w || Bring the template and the image to a uniform scale: subtract the mean and divide by the variance h( x, y ) f ( x, y ) http://scien.stanford.edu/pages/labsite/2008/psych221/projects/08/MariaJabon/index.htm The University of Neighborhood Processing (filtering) Template Matching Normalized cross-correlation Ontario f w f w ( x, y ) arccos h fw || h |||| f w || Extremely time consuming due to sliding window g cos( ) f | g | 1 h Slide credit: Matlab manual The University of Neighborhood Processing (filtering) Template Matching Ontario g cos( ) h f Slide credit: OpenCV manual