Report

Computer Vision, Robert Pless Lecture 2, Pinhole Camera Model Last time we talked through the pinhole projection. • This time we are going to look at the coordinate systems in the pinhole camera model. Computer Vision, Robert Pless Some context Our goal is to understand the process of stereo vision, which requires understanding: • How cameras capture the world • Representing where cameras are in the world (and relative to each other) • Finding similar points on each image Computer Vision, Robert Pless “Normalized Camera” • Pinhole at (0,0,0), point P at (X,Y,Z) • Virtual film at Z = 1 plane. • Point X,Y,Z is imaged at intersection of: – Line from (0,0,0) to (X,Y,Z), and – the Z = 1 plane X P Y Z • Intersection point (x,y), has coordinates – (X/Z, Y/Z, 1) • This is called the “normalized camera model” • What happens when focal length is not 1? Computer Vision, Robert Pless x p y Camera Center of Projection Pinhole, (Center of projection), at (0,0,0) (X,Y,Z) f Z-axis Image plane Computer Vision, Robert Pless …but the cow is not a sphere We don’t just want to know where, mathematically, the point ends up on the image plane… we want to know the pixel coordinates of the point. Why might this be hard?! Computer Vision, Robert Pless Intrinsic Camera Parameters The pixels may be rectangular: x = aX/Z Y = bY/Z CCD sensor array Computer Vision, Robert Pless Intrinsic Camera Parameters Chip not centered, or 0,0 may not be center of the chip. x = aX/Z + x0 Y = bY/Z + y0 CCD sensor array Computer Vision, Robert Pless Intrinsic Camera Parameters cheap CMOS chip cheap lens image cheap glue cheap camera x = aX/Z – a cot(q) Y/Z + x0 y= b/sin(q) Y/Z + y0 Computer Vision, Robert Pless Ugly relationship between (X,Y,Z) and (x,y). Homogeneous Coordinates • Represent 2d point using 3 numbers, instead of the usual 2 numbers. • Mapping 2d 2d homogeneous: – (x,y) (x,y,1) – (x,y) (sx, sy, s) • Mapping 2d homogeneous to 2d – (a,b,w) (a/w, b/w) • The coordinates of a 3D point (X,Y,Z) *already is* the 2D (but homogeneous) coordinates of its projection. • Will make some things simpler and linear. Computer Vision, Robert Pless Why?! x = aX/Z – a cot(q) Y/Z + x0 y= b/sin(q) Y/Z + y0 x a y 0 1 0 a cot (q ) b sin(q ) 0 x0 X y0 Y 1 Z Computer Vision, Robert Pless x fx y 0 1 0 s fy 0 x0 X y0 Y 1 Z Homogenous Coordinates! • Equation of a line on the plane? • Equation of a plane through the origin in 3D? Computer Vision, Robert Pless x fx y 0 1 0 s fy 0 x0 X y0 Y 1 Z Given enough examples of 3D points and their 2D projections, we can solve for the 5 intrinsic parameters… Example calibration object… Computer Vision, Robert Pless Practical Linear Algebra 1: • Given many examples of the equation at the right, solve for the matrix of values fx, fy, s, x0, y0 x fx y 0 1 0 • How can we solve for this? Computer Vision, Robert Pless s fy 0 x0 X y0 Y 1 Z … and all this assumes that we know X,Y,Z x fx y 0 1 0 s fy 0 x0 X y0 Y 1 Z Given enough examples of 3D points and their 2D projections, we can solve for the 5 intrinsic parameters… Exactly Known! (0,0,0) Computer Vision, Robert Pless x fx y 0 1 0 s fy 0 x0 X y0 Y 1 Z More commonly, we can put a collection of points (usually in a grid) in the world with known relative position, but with unknown position relative to the camera. Then we need to solve for the intrinsic calibration parameters, and the extrinsic parameters (the unknown relative position of the grid). How do we represent the unknown relative position of the grid? Computer Vision, Robert Pless The grid defines its own coordinate system That coordinate system has a Euclidean transformation that Z relates its coordinates to the camera coordinates… that is, there is some R, T such that: Y X X grid X Y R Ygrid T Z Z grid With enough example X grid X . . . Tx points, could we solve for Ygrid Y . R .Ty R,T? Z Z . . . T grid z 1 ComputerVision, Robert Pless An aside…Representation of Rotation with a Rotation Matrix Rotation matrix R has two properties: Property 1: R is orthonormal, i.e. In other words, Property 2: R RI T R T R 1 Determinant of R is 1. Therefore, although the following matrix is orthonormal, it is not a rotation matrix because its determinant is -1. 1 0 0 0 1 0 0 0 1 this is a reflection matrix Computer Vision, Robert Pless Representation of Rotation -- Euler Angles • Euler angles • pitch: rotation about x axis : • yaw: rotation about y axis: • roll: rotation about z axis: Z X ~ X C cos sin 0 cos 0 sin 1 0 ~ C Y sin cos 0 1 0 0 cos ~ C sin 0 cos 0 sin Z 0 0 1 Computer Vision, Robert Pless 0 X W W sin Y cos Z W Y Putting it all together… x fx y 0 1 0 x fx y 0 1 0 s fy 0 s fy 0 x0 X y0 Y 1 Z + X grid X . . . Tx Ygrid Y . R .Ty Z . . . T Z grid z 1 X grid x0 . . .Tx Ygrid y0 . R .Ty Z grid 1 . . . Tz 1 X grid x Ygrid y P 1 Z grid 1 P is a 4X3 matrix, that defines projection of points (used in graphics). P has 11 degrees of freedom, (the scale of the matrix doesn’t matter). 5 intrinsic parameters, 3 rotation, 3 translation Computer Vision, Robert Pless • Given many examples of: – World points (X,Y,Z), and – Their image points (x,y) • Solve for P. • Then . . . Tx P . KR .Ty . . .T z Rotation + intrinsic, all mixed up. translation Computer Vision, Robert Pless X world x Yworld y P Z world 1 1 . . . Tx P . KR .Ty . . .T z Then take the QR decomposition of this part of the matrix to get the rotation and the intrinsic parameters. Computer Vision, Robert Pless (from mathworld). Computer Vision, Robert Pless Other common trick, use many planes, then you have to solve for multiple different R,T, one for each plane… Counting: For each grid, we have 3 unknowns (translation), 3 unknowns (rotation). + 5 extra unknowns defining the intrinsic calibration. Method. For a collection of planes, calculate the homography between the image and the real world plane. Computer Vision, Robert Pless Computer Vision, Robert Pless Computer Vision, Robert Pless Computer Vision, Robert Pless Computer Vision, Robert Pless Computer Vision, Robert Pless Corners.. Computer Vision, Robert Pless Computer Vision, Robert Pless Computer Vision, Robert Pless What is left? Barrel and Pincushion Distortion wideangle tele Computer Vision, Robert Pless What other calibration is there? Computer Vision, Robert Pless Models of Radial Distortion x xd 2 4 (1 k1r k 2 r ) y yd distance from center Computer Vision, Robert Pless More distortion. Computer Vision, Robert Pless Distortion Corrected… Computer Vision, Robert Pless Applications? Computer Vision, Robert Pless Cheap applications x fx y 0 1 0 s fy 0 X grid x0 . . .Tx Ygrid y0 . R .Ty Z grid 1 . . . Tz 1 X grid x Ygrid y P 1 Z grid 1 • Can you use this? If you know the 3D coordinates of a virtual point, then you can draw it on your image… • Often hard to know the 3D coordinate of a point; but there are some (profitable) special cases… Computer Vision, Robert Pless Applications First-down line courtesy of Sportvision Computer Vision, Robert Pless Applications Virtual advertising courtesy of Princeton Video Image Computer Vision, Robert Pless Recap Three main concepts: 1) Projection, 2) Homogeneous coordinates 3) Camera calibration matrix Computer Vision, Robert Pless Computer Vision, Robert Pless Another Special Case, the world is a plane. • Projection from the world to the image: x fx y 0 1 0 s fy 0 x0 y0 1 R X grid Tx Ygrid Ty Z grid Tz 1 0 Irrelevant • Ignore z-coordinate (it is 0 anyway), drop the 3rd column of the P matrix, then you get a mapping between the plane and the image which is an arbitrary 3 x 3 matrix. This is called a “homography”. Computer Vision, Robert Pless Homography: (x’,y’,1) ~ H (x,y,1) • Not a constraint. Unlike the fundamental matrix, tells you exactly where the point goes. • Point (x,y) in one frame corresponds to point (x’,y’) in the other frame. If we need to think about multiple points, we may put subscripts on them. • Being careful about the homogenous coordinate, we write: x wx ' H y wy ' 1 w Computer Vision, Robert Pless Homography is a “simple” example of a 3D to 2D transformation It is also the “most complicated” linear 2D to 2D transformation. What other 2D 2D transformations are there? Computer Vision, Robert Pless Homography is most general, encompasses other transformations Projective 8 dof Affine 6 dof Similarity 4 dof Euclidean 3 dof h11 h12 h 21 h22 h31 h32 a11 a12 a 21 a22 0 0 h13 h23 h33 tx t y 1 sr11 sr12 t x sr sr t 22 y 21 0 0 1 r11 r12 t x r r t 21 22 y 0 0 1 Computer Vision, Robert Pless Views of a plane from different viewpoints, any view of a scene from the same viewpoint. Images of a “far away” object under any rotation Camera looking at an assembly line w/ zoom. Camera looking at an assembly line. Invariants… Projective 8 dof h11 h 21 h31 Affine 6 dof a11 a 21 0 Similarity 4 dof sr11 sr 21 0 Euclidean 3 dof r11 r 21 0 h12 h22 h32 a12 a22 0 sr12 sr22 0 r12 r22 0 h13 h23 h33 Concurrency, collinearity, order of contact (intersection, tangency, inflection, etc.), cross ratio tx t y 1 Parallellism, ratio of areas, ratio of lengths on parallel lines (e.g midpoints). tx t y 1 tx t y 1 Ratios of any edge lengths, angles. Absolute lengths, areas. Computer Vision, Robert Pless Image registration Determining the 2d transformation that brings one image into alignment (registers it) with another. Computer Vision, Robert Pless Image Warping • What kind of warps are these? 0.6101 0.1918 34.2411 0.1680 0.6000 26.7034 0.0011 0.0010 1 Computer Vision, Robert Pless How to solve for these mappings? Given: Solve for: 0.6101 0.1918 34.2411 0.1680 0.6000 26.7034 0.0011 0.0010 1 Computer Vision, Robert Pless Unwrapping a matrix. x wx ' H y wy ' 1 w a d g b e h c x wx ' f y wy ' 1 1 w Write out the lines of this matrix equation. ax by c wx' dx ey f wy ' gx hy 1 w And remember whichComputer variables are unknown. Vision, Robert Pless Unwrapping a matrix. a d g b e h c x wx ' f y wy ' w 1 1 ax by c wx ' i dx ey f wy ' i gx hy 1 w ax by c 0d 0e 0 f 0g 0h wxi' 0a 0b 0c dx ey f 0g 0h wyi' 0a 0b 0c 0d 0e 0 f gx hy 1 w ax by c 0d 0e 0 f 0g 0h wx' 0 0a 0b 0c dx ey f 0g 0h wy ' 0 0a 0b 0c 0d 0eComputer 0 fVision, gx hy w 1 Robert Pless a d g b e h c x wx ' wy ' f y 1 1 w ax by c 0d 0e 0 f 0g 0h wx' 0 0a 0b 0c dx ey f 0g 0h wy ' 0 0a 0b 0c 0d 0e 0 f gx hy w 1 a Looks like another matrix equation: b c ' x y 1 0 0 0 0 0 xi d 0 0 ' 0 0 0 x y 1 0 0 y e i 0 0 0 0 0 0 x y 1 f 1 g h w Computer Vision, Robert Pless a d g b e h c x wx ' wy ' f y 1 1 w ax by c 0d 0e 0 f 0g 0h wx' 0 0a 0b 0c dx ey f 0g 0h wy ' 0 0a 0b 0c 0d 0e 0 f gx hy w 1 Looks like another matrix equation: Data from different points x 0 0 x 0 0 x 0 0 x 0 0 0 0 0 a 0 0 x y 1 0 0 yi' 0 0 0 b 0 0 0 0 0 x y 1 0 0 0 c 1 ' 1 0 0 0 0 0 0 xi 0 0 d 0 0 x y 1 0 0 0 yi' 0 0 e 0 0 0 0 0 x y 0 1 0 0 f 1 ' 0 1 0 0 0 0 0 0 0 xi 0 g 0 x y 1 0 0 0 0 yi' 0 h 0 0 0 0 0 x y 0 0 1 0 w 1 1 0 0 0 0 0 0 0 0 xi' w2 0 ' 0 x y 1 0 0 0 0 0 yi w3 0 0Computer 0 0Vision, 0 Robert x yPless0 0 0 1 w4 1 y 1 0 0 0 0 0 0 0 y 0 0 y 0 0 y 0 0 xi' a d g b e h c x wx ' wy ' f y 1 1 w Why does this suck? • Maybe you have error in finding corresponding points, and want to use many many corresponding points. Then your number of unknowns keeps growing… • Is there a better way? • What is x’? Computer Vision, Robert Pless The game of “finding the linear constraint…” x’ = wx’ / w ax by c wx gx hy 1 w a d g ' b e h ax by c x gx hy 1 ' x' ( gx hy 1) ax by c Computer Vision, Robert Pless c x wx ' f y wy ' 1 1 w Non-linear Linear (in a,b,c,g,h) x' ( gx hy 1) ax by c gxx' hyx' x ' ax by c gxx hyx ax by c x ' ' ' ax by c gxx' hyx' x ' ax by c 0d 0e 0 f gxx' hyx' x ' 0a 0b 0c dx ey f gxy hyy y ' Computer Vision, Robert Pless ' ' ax by c 0d 0e 0 f gxx i' hyx i' xi' 0a 0b 0c dx ey f gxy i' hyy i' yi' 0 0 xx' x y 1 0 0 0 0 x y 1 xy' yx' yy' a b c d x' e y ' f g h And just add two more rows for each corresponding point Computer Vision, Robert Pless 0 0 xx' x y 1 0 0 0 0 x y 1 xy' yx' yy' a b c d x' e y ' f g h Ax=b Matlab: x = A\b Then make your homography matrix by rearranging x into a 3 x 3 matrix Size of A? b? Computer Vision, Robert Pless Computer Vision, Robert Pless H12 H23 H34 (H12)-1 (H34H23H12) (H34H23H12)-1= ((H12)-1(H23)-1(H34)-1) Chaining, inverses, and pre-computation can all be done on the matrices… Computer Vision, Robert Pless Computer Vision, Robert Pless More than Geometry: image processing • The above gave different ways of transforming the coordinates of one image into the coordinates of another. Graphics question. – Given one image, the homography H, make the second image. • (why would you want to do this? One application is image tweening…). – Process: • Take pixel (x,y) of image 1, and put its color at pixel H(x,y,1) in new image. Repeat. – Problems? Computer Vision, Robert Pless Backward Transformation • Because of the above, some techniques estimate the parameters of the “backwards” transformation, H-1: – From the destination image to the source image • Remember, you don’t have to work hard to find the backwards transformation. It’s just the matrix inverse. Computer Vision, Robert Pless Mapping Pixel Values - Going Backwards p = H-1(p’) p’ “Source” image, Is “Destination” image, Id Id(p’) = Is(H-1(p’)) Intensity at destination pixel p’ = the intensity at the source pixel H-1(p’) Computer Vision, Robert Pless If you project backward from a real pixel, you also won’t land exactly on a pixel??!!? (but there is a good way of making up a value) H-1(p’) Pixel centers… Computer Vision, Robert Pless If you project backward from a real pixel, you also won’t land exactly on a pixel??!!? (but there is a good way of making up a value) Computer Vision, Robert Pless Use linear interpolation along the top and bottom horizontal lines to determine IA and IB: IA = (1 – f)•I(xi, yj) + f•I(xi+1, yj) IB = (1 – f)•I(xi, yj+1)+ f•I(xi+1, yj+1) Use linear interpolation along the vertical line between IA and IB to determine I(x,y): I(x,y) = (1 – g)•zA + g•zB Or, I(x,y) = (1 – f)•(1 – g)•I(xi, yj) (1 – f)• g •I(xi, yj+1) + + f•(1 – g)•I(xi+1, yj) + f• g •I(xi+1, yj+1) Computer Vision, Robert Pless Beyond geometry… Computer Vision, Robert Pless