Report

Motion Estimation I What affects the induced image motion? • Camera motion • Object motion • Scene structure Example Flow Fields • This lesson – estimation of general flow-fields • Next lesson – constrained by global parametric transformations The Aperture Problem So how much information is there locally…? The Aperture Problem Not enough info in local regions Copyright, 1996 © Dale Carnegie & Associates, Inc. The Aperture Problem Not enough info in local regions Copyright, 1996 © Dale Carnegie & Associates, Inc. The Aperture Problem Copyright, 1996 © Dale Carnegie & Associates, Inc. The Aperture Problem Information is propagated from regions with high certainty (e.g., corners) to regions with low certainty. Copyright, 1996 © Dale Carnegie & Associates, Inc. Such info propagation can cause optical illusions… Illusory corners • Direct (intensity-based) Methods 1. Gradient-based (differential) methods (Horn &Schunk, Lucase & Kanade) 2. Region-based methods (Correlation, SSD, Normalized correlation) • Feature-based Methods Brightness Constancy Assumption Image I (taken at time t+1) x u, y v J x, y Image J (taken at time t) x, y u, v I x u , y v The Brightness Constancy Constraint Brightness Constancy Equation: J ( x , y ) I ( x u ( x , y ), y v ( x , y )) Linearizing (assuming small (u,v)): J ( x, y ) I ( x, y ) I x ( x, y ) u ( x, y ) I y ( x, y ) v( x, y ) 0 I x ( x, y ) u ( x, y ) I y ( x, y ) v( x, y ) I ( x, y ) J ( x, y ) I x ( x, y ) u ( x, y ) I y ( x, y ) v( x, y ) I t ( x, y ) u ( x , y ) I ( x, y ) I t ( x, y ) v( x, y ) T Observations: * One equation, 2 unknowns * A line constraint in (u,v) space. * Can recover “Normal-Flow” = (the component of the flow in the gradient direction) Need additional constraints… Horn and Schunk (1981) Add global smoothness term E I u I yv It 2 x u 2 x u 2 y v 2 x v ( x, y) (x,y) Error in brightness constancy equation Minimize: Smoothness error Ec Es Solve by using calculus of variations 2 y Horn and Schunk (1981) Inherent problems: * Smoothness assumption wrong at motion/depth discontinuities over-smoothing of the flow field. * How is Lambda determined…? Lucas-Kanade (1984) Assume a single displacement (u,v) for all pixels within a small window (e.g., 5x5) Geometrically -- Intersection of multiple line constraints Algebraically -- E (u , v ) Minimize E(u,v): I ( x , y )W indow ( x , y )u I y ( x , y )v I t 2 x Lucas-Kanade (1984) Minimize E(u,v): E (u , v ) I ( x , y )u I y ( x , y )v I t 2 x ( x , y )W indow Differentiating w.r.t u and v and equating to 0: I x2 I x I y u I x I t 2 I I I v x y y I y I t I I T U I It Solve for (u,v) [ Repeat this process for each and every pixel in the image ] Singularites I I x I y 2 x I I x y 2 I y Where in the image will this matrix be invertible and where not…? Homework Linearization approximation iterate & warp estimate update Initial guess: Estimate: x0 x Linearization approximation iterate & warp estimate update Initial guess: Estimate: x0 x Linearization approximation iterate & warp estimate update Initial guess: Estimate: x0 x Linearization approximation iterate & warp x0 x Revisiting the small motion assumption Is this motion small enough? Probably not—it’s much larger than one pixel (2nd order terms dominate) How might we solve this problem? Coarse-to-Fine Estimation Advantages: (i) Larger displacements. (ii) Speedup. (iii) Information from multiple window sizes. iterate + u I x u I y v It 0 u refine u=1.25 pixels u=2.5 pixels Δu ==> small u and v ... u=5 pixels image J Pyramid of image J u=10 pixels image I Pyramid of image I Optical Flow Results Optical Flow Results Lucas-Kanade (1984) Inherent problems: * Still smoothes motion discontinuities (but unlike Horn & Schunk, does not propagate error across the entire image) * Local singularities (due to the aperture problem) • Maybe increase the aperture (window) size…? • But no longer a single motion… Global parametric motion estimation – next week. Motion Magnification Wu, Rubinstein, Shih, Guttag, Durand, Freeman “Eulerian Video Magnification for Revealing Subtle Changes in the World”, SIGGRAPH 2012 Source video: baby.mp4 Result: baby-iir-r1-0.4-r2-0.05-alpha-10-lambda_c-16-chromAtn-0.1.mp4 Paper + videos can be found on: http://people.csail.mit.edu/mrub/vidmag Motion Magnification Could compute optical flow and magnify it But… very complicated (motions are almost invisible) Alternatively: ∙ + ∙ = - ∙ •s + ∙ •s = - •s But holds only for small u•s and v•s apply coarse to fine to generate larger motions Motion Magnification What is •s equivalent to? (, ) (time) This is equivalent to keeping the same temporal frequencies, but magnifying the amplitude (increase frequency coefficient). Can decide to do this selectively to specific temporal frequencies (e.g., a range of frequencies of expected heart rates). Motion Magnification Wu, Rubinstein, Shih, Guttag, Durand, Freeman “Eulerian Video Magnification for Revealing Subtle Changes in the World”, SIGGRAPH 2012 Paper + videos can be found on: http://people.csail.mit.edu/mrub/vidmag A simplified version of this work the next programming exercise • Exercise will be posted within a few days • Meanwhile, please read SIGGRAPH’2012 paper