TETRA-project: SMART DATA CLOUDS (2014 – 2016) . Flemish Agency for Innovation by Science and Technology Contact persons: [email protected] [email protected] [email protected] Website: https://www.uantwerpen.be/op3mech/ TETRA: Smart Data Clouds (2014-2016): a. Security, track and trace b. Traffic control and classifications c. (Smart) Navigation d. Dynamic 3D body scan industrial cases [Haven van Antwerpen]. [Macq, LMS, Melexis] [Wheelshairs, PIAM, mo-Vis, Automotive LMS] [RSscan, ICRealisations, ODOS] Data Fusion: ind. Vision linked with CAE. Data from RGB, IR, ToF and HS cameras. TETRA: Smart Data Clouds (2014-2016): a. Wheelchair control - navigation: b. Gesture & Body-analysis: healthcare applications [mo-Vis, PIAM, Absolid] [RSScan, ICrealisation, Phaer, ODOS] ‘Data Fusion’ usable for future Healthcare . 03D2xx-cameras IFM-Electronics 64x50 pix. PMD[Vision]CamCube 3.0 352 x 288 pix! Recent: pmd PhotonICs® 19k-S3 Swiss Ranger 4500 MESA 176 x 144 pix. BV-ToF 128x120 pix. DepthSense 325 ( 320 x 240 pix ) Fotonic RGB_C-Series 160 x 120 pixel Previous models: P – E series. Melexis: EVK75301 80x60 pix. Near Future: MLX75023 automotive QVGA ToF sensor . ODOS: 1024x1248 pix. Real.iZ-1K vision system ToF VISION: world to image, image to world - conversion 1 1 P = [ tg(φ) tg(ψ) 1 d/D ] , j J/2 i P’ = [ tg(φ) tg(ψ) 1 ] . ( f can be chosen to be the unit.) j0 J φ i0 I/2 (0,0,f) f=1 z I x y ψ R A u.ku r D v.kv d D/d = x/uk = y/vk = z/f Every world point is unique with respect to a lot important coordinates: x, y, z, Nx, Ny, Nz, kr, kc, R, G, B, NR , NG , NB , t° , t ... The basis of our TETRA-project: ‘Smart Data Clouds’ kr ,kc N u = j – j0 ; uk = ku*u v = i – i0 ; vk = kv*v tg φ = uk/f tg ψ = vk/f r = √(uk²+f²) d = √(uk²+vk²+f²) D/d-related calculations. (1) For navigation purposes, the free floor area can easily be found from: di/Di = e/E = [ f.sin(a0) + vi.cos(a0) ] / E = [ tg(a0) + tg(ψi) ].f.cos(a0)/E . Since (d/D)i = f /zi this is equivalent with: zi . [ tg(a0) + tg(ψi) ] = E/cos(a0) . Camera sensor Camera inclination = a0 . f di e ψi vi Di Camera bounded parallel to the floor. a0 E zi Floor. D/d-related calculations. (2) Fast calculations !! 1 D The world normal vector n , at a random image position (v,u) . 4 3 d v d² = u² + v² + f². O n u f nx ~ f.(D4/d4 - D3/d3)/(D4/d4 + D3/d3) = f.(z4 – z3)/(z4 + z3) ny ~ f.(D2/d2 - D1/d1)/(D2/d2 + D1/d1) = f.(z2 – z1)/(z2 + z1) nz ~ - (u.nx + v.ny + 1 ) ▪ 2 Coordinate transformations yPc 1. Camera x // World x // Robot x 2. World (yw , zw) = Robot (yr , zr) + ty vt 3. yPw = - (z0 – zPc).sin(a) + yPc.cos(a) = yPr + ty f 4. zPw = (z0 – zPc).cos(a) + yPc.sin(a) = zPr a zw P zPc A camera re-calibration for ToF cameras is easy and straightforward !! zr a yw Work plane = reference. yr ty Pepper handling First image = empthy world plane Next images = random pepper collections. Connected peppers can be distinguished by means of local gradients. Gradients can easily be derived from D/d-ratios. Thickness in millimeter Calculations are ‘distance’ driven, x, y and z aren’t necessary. Fast calculations ! YouTube: KdGiVL Bin picking & 3D-OCR Analyse ‘blobs’ one by one. Find the centre of gravity XYZ, the normal direction components Nx, Ny, Nz , the so called ‘Tool Centre Point’ and the WPS coordinates. Beer barrel inspection. IDS uEye UI-1240SE-C O3D2xx x y x y z ToF - RGB correspondency vc,P/F – kv.vt/f = tx.√(z²P+y²P) uc/F = ku.ut/F . MESA SR4000 RGBd DepthSense 311 1. Find the z –discontinuity 2. Look for vertical and forward oriented regions 3. Check the collineraity 4. Use geometrical laws in order to find x, y, z and b. ToF IFM O3D2xx. 1. Remove weak defined pixels. 2. Find the z –discontinuity 3. Look for vertical and forward oriented regions 4. Check the collineraity 5. Use geometrical laws in order to find x, y, z and b. CamBoard Nano Basic tasks of ToF cameras in order to support Healthcare Applications: • Guide an autonomous wheelchair along the wall of a corridor. • Avoid collisions between an AGV and unexpected objects. • Give warnings about obstacles (‘mind the step’…, ‘kids’, stairs…) • Take position in front of a table, a desk or a TV screen. • Drive over a ramp at a front door or the door leading to the garden. • Drive backwards based on a ToF camera. • Automatic parking of a wheelchair (e.g. battery load). • Command a wheelchair to reach a given position. • Guide a patient in a semi automatic bad room. • Support the supervision over elderly people. • Fall detection of (elderly) people. Imagine a wheelchair user likes to reach 8 typical locations at home: 1. Sitting on the table. 2. Watching TV. 3. Looking through the window. 4. Working on a PC. 5. Reaching the corridor 6. Command to park the wheelchair. 7. Finding some books on a shelf. 8. Reaching the kitchen and the garden. ToF guided navigation for AGV’s. D2 = DP ? Instant Eigen Motion: translation. Distance correspondence during a pure translation + horizontal camera. d1 = sqrt(u1² + v1² + f²) RMS error. x1 = u1* D1/d1; y1 = v1* D1/d1; z1 = f* D1/d1 Distance correspondence during a pure translation + horizontal camera. D22 = D1² + δ² - 2.δ.D1.cos(A) D22 = D1² + δ² - 2.δ.z1 u2/f = x2/(z1 - δ); δ ( x2 = x1 ! ) For every δ there is a D2 . For one single δ the error D2 – DP is minimal. v2/f = y1/(z1 - δ) . Measurement at (v2,u2) DP . δ d1 Random world f f v1,u1 v2,u2 D1 A y1 D2 = DP ? P Every Random point P can be used in order to estimate de value δ. More points make a statistical approach possible. Random points P! (In contrast: Stereo Vision must find edges, so texture is pre-assumed ) ToF guided Navigation of AGV’s. Instant Eigen Motion: planar rotation. D2 = DP ? Image data: tg(β1) = u1/f ; tg(β2) = u2/f . Next sensor position P Here: x<0 R>0. D1 z1 Previous sensor position Task: find the correspondence β1 β2 ; Procedure: With 0 < |α| < α0 z2 D2 DP With |x| < x0 Projection rules for a random point P : β2 β1 α xP 2 D2 .sin(2 ) D1.sin( 1 ) x.cos( ) z P 2 D2 .cos(2 ) D1.cos( 1 ) x.sin( ) R+ tan( 2 ) δ1 α x Parallel processing possible! D1.sin( 1 ) x.cos( ) D1.cos( 1 ) x.sin( ) D2² = xP2² + zP2² D22 D12 x2 2 2.D1.sin(1) 2. D1.cos( 1 ) x.sin( ) ToF guided navigation of AGV’s. Instant Eigen Motion: planar rotation. Previous image Next image White spots are random selected points Minimum RMS = 9.4211 e.g. Make use of 100 Random points. Minimum RMS = 6.4 R = 1152 alfa = 0.24 ratio = 1 Result = Radius & Angle 30 20 D2,i = DP,i ? 10 0 60 40 20 0 Value x. 0 20 40 60 80 Angle A. CPUtime = 0.11202 100 Research FTI Master Electromechanics Info: [email protected] ; Conrad: DJI Phantom RTF Quadrocopter [email protected] Research: ToF driven Quadrocopters Combinations with IR/RGB Security flights over industrial areas. TETRA-project 2014-2016 ‘Smart Data Clouds’ Quadrocopter navigation based on ToF cameras z_world xc P zc_camera α xc BV-ToF 128x120 pix. zc0 x1 x-world The target is to ‘hover’ above the end point of a black line. If ‘yaw’ is present it should be compensated by an overall torque moment. Area seen by the camera Camera Horizon Q(x1,y1) (vc,uc)P(x,y,z) P(x,y) t (Vc,Uc) t Q(x1,y1) Meanwhile the translation t can be evaluated. The global effect of the roll and pitch angles represent themselves by means of the points P and Q. The actual copter speed v is in the direction PQ. At the end P and Q need to come together without oscillations, while |t| becomes oriented in the ydirection. _world y_world An efficient path can be followed up by means of Fuzzy Logic principals.