Report

UMR 5205 On the Efficiency of Image Metrics for Evaluating the Visual Quality of 3D Models Guillaume Lavoué Université de Lyon LIRIS Mohamed Chaker Larabi Université de poitier XLIM-SIC Libor Vasa University of West Bohemia An illustration Original Watermarking Wang et al. 2011 Smoothing Taubin, 2000 0.14 Watermarking Cho et al. 2006 0.51 Simplification Lindstrom, Turk 2000 0.40 Noise addition 0.62 Same Max Root Mean Square Error (1.05 × 10-3) 0.84 Quality metrics for static meshes Local curvature statistics MSDM [Lavoué et al. 2006] MSDM2 [Lavoué 2011] [Torkhani et al. 2012] Distorted model Local differences of statistics Matching Original model Local Distortion Map Spatial pooling Global Distortion Score 3 Our previous works Distortion score Why not using Image Quality Metrics? Such image-based approach has been already used for driving simplification [Lindstrom, Turk, 2000][Qu, Meyer, 2008] 4 Our study Determine the best set of parameters to use for such imagebased quality assessment approach. Compare this approach to the most performing model-based metrics. 5 Many parameters Which 2D metric to use? How many views, which views? How to combine the 2D scores? Which rendering, lighting? In our study, we consider: o 6 image metrics o 2 rendering algorithms Around 100,000 images o 9 lighting conditions o 5 ways of combining image metric results o 4 databases to evaluate the results 6 Image Quality Metrics Simple PSNR and Root Mean Square Error MSSIM (multi-scale SSIM) [Wang et al. 2003] VIF (visual information fidelity) [Sheikh and Bovik, 2006] IWSSIM (information content weighted SSIM) [Wang and LI, 2011] FSIM (feature similarity index) [Zhang et al. 2011] State of the art algorithms 7 Generation of 2D views and lightning conditions 42 cameras placed uniformly around the object Rendering using a single white directional light source The light is either fixed with respect to the camera, or with respect to the object 3 positions: front, top, top-right So we have 3*2 = 6 lighting conditions We also consider averages of object-light, camera-light and global 9 conditions 8 Image Rendering Protocols We consider 2 ways of computing the normals, with or without averaging on the neighborhood. 9 Pooling algorithms How to combine the per-image quality score into a single one? Minkowski norm is popular: We also consider image importance weights [Secord et al. 2011] Perceptual model of viewpoint preference Surface visibility 10 The MOS databases The LIRIS/EPFL General-Purpose Database 88 models (from 40K to 50K vertices) from 4 reference objects. Non uniform noise addition and smoothing. The LIRIS Masking Database 26 models (from 9K to 40K vertices) from 4 reference objects. Noise addition on smooth or rough regions. The IEETA Simplification Database 30 models (from 2K to 25K vertices) from 5 reference objects. Three simplification algorithms. The UWB Compression database 68 models from 5 reference objects Different kinds of artefacts from compression 11 Results and analysis Basically we have a full factorial experiments heavily used in statistics to study the effect of different factors on a response variable We consider 4 factors: o The metric (6 possible values) o The lighting (9 possible values) o The pooling (5 possible values) o The rendering (2 possible values). 540 possible combinations We consider two response variables: o Sperman correlation over all the objects o Sperman correlation averaged per objects 12 Results and analysis For a given factor associated with n possible values, we have n sets of paired spearman coefficients. To estimate the effect of a given factor on the objective metric performance, we conduct pairwise comparisons of each of its value between the others (i.e. n(n-1)/2 comparisons). We have paired values, so we can do better than a simple comparison of the means. Statistical significance test (not Student but Wilcoxon signed rank test). We study the median of paired differences, as well as the 25th and 75th percentiles. 13 Influence of the metrics IWSSIM provides the best results FSIM and MSSIM are 2nd best, significantlky better than MSE and PSNR. VIF provides instable results (see the percentiles). 14 Influence of the lighting Indirect illuminations provide better results Light has to be linked to the camera Object-front is not so bad, but not its performances are not stable. 15 Influence of the pooling Low values of P are better. Weights do not bring significant improvments. 16 Comparisons with 3D metrics For easy scenarios: 2D metrics are excellent However when the task becomes more difficult, 3D metrics are better But, still, simple image-based metrics are better than simple geometric ones. 17