### Slides

```Outdoor Image
Processing
1
Photometric stereo for outdoor webcams

"Photometric stereo for outdoor webcams"
Ackermann, J.; Langguth, F.; Fuhrmann, S.; Goesele, M.; , CVPR 2012
Overview:

Photometric stereo from time lapse video captured over a long time span.

Retrieves

Surface Normals

Basic Materials

Material Mixtures

Indirect light
2
Assumptions

GPS location of the camera, object and sky mask, per image time stamp are
available
3
4
Selecting Subsets of images

Image Filtering:
1) Discard images with 10% of the image or the object is overexposed
2) Select only daytime images, zenith < 85 degrees
3) Discard bad weather images – select only top 50% according to score:
SI= Isky + IObj
Isky = median of sky pixel intensities
IObj = 75th percentile of object pixel intensities
5
Selecting Subsets of images

Two Image subsets required;
1) Clear sky images for camera calibration,
2) Images with good weathers and well illuminated object for photometric stereo
Iteratively select required number of images by updating penalty using a 2D
Gaussian function and selecting the best image at that iteration
6
Obtaining light direction


Camera Calibration:
1) Radiometric response obtained using Kim et al. ( uses pixels under the same lighting
conditions to solve for the response function).
2) Absolute zenith, azimuth of the Sun obtained using cam location, timestamp.
3) Use the sky as calibration target ( Lalonde et al.) to find camera zenith, azimuth


Imax,p / Imin,p < 1.4

Otherwise, Ii,p < 1.5*median10% darkest pixels => shadowed in Ii
End of Stage 1 ( obtain subset of images with light direction)
7
Photometric Stereo stage

Intensity of image i at pixel p and channel c, Ii,p,c = Isun,i,p,c + Isky,I,p,c

Reflectance at a pixel is linear combination of basis materials, fm,c
Sun light model
Intensity of the sun
Surface normal
Material mixing coeff
Portion of sky visible at p
Sun direction
8
Light Model

Sky light model
Assume
Finally,
Optimize for li,c, fm,c, np, γp,m .
Vp is replaced by using images Ip s.t. pixel p is not in shadow
9
Initialization

Set Sp,c to zero , assume constant light intensities and Lambertian scene
1.
Obtain initial estimates for surface normals and albedo
2.
Use these to find initial estimates of the light intensities
3.
Cluster the albedos to get an initialization of material properties at each pixel
1) Surface normal and albedo

Solve for classical photometric stereo
10
Initialization

2) Relative light intensities:


Need six surface points with similar albedo and differing normal

1) Cluster albedos in 4 groups

2) Cluster normals for pixels with the most frequent albedo

3) Pick normal from different clusters
3) Initial material estimation:

Cluster albedos in sRGB

Identify pure pixel sets for each of the fundamental materials

Solve for the BRDF parameters
11
Iterative Refinement

Intensity estimate:
updated in each following step
1) Material Fitting

Find optimal parameters for all materials simultaneously, not for only pure pixels

Minimize
12
Iterative Refinement

Light intensity optimization:


Minimize
,
Material and normal map optimization:

Minimize

Material parameters, light intensities are fixed, only normals, sky light at each pixel, and
material mixing coeff are optimized
13

Single-Image Shadow Detection and Removal using Paired Regions
Ruiqi Guo, Qieyun Dai, Derek Hoiem. CVPR 2012

Employs a region based approach.

Perform pairwise classification (of illumination conditions) of regions based on
appearance.

Graph cut is used for the labeling.

Soft matting for refinement

14

Maximize

cishadow – single region classifier confidence * region area

cijdiff , cijdiff – pairwise classifier confidence * f(region areas)

y – shadow labels for regions
15


Single region classifier (with χ2 kernel) features
1.
Color histograms
2.
Texton histograms
Internal appearance of a given region is not enough
Comparison between regions of same material needed
Pairwise region classifier (RBF kernel) features
1.
Χ2 distance between color and texton histograms
2.
Ratios of RGB average intensity ( ρr = Ravg1 /Ravg2, …)
3.
Chromatic alignment (ρr/ρg)
4.
Normalized distances between the regions
16
Pairwise region graph

Different illumination
 black-white

Same-illumination
 Green

Not related
 Orange
17
Pairwise region graph
18
Apply Graph cut

Reformulate the cost function to apply Graph cut
19
Solution:
20
Slide from Guo et al.

Simple light model: direct light + env. Light

Relighting:
Estimate how much direct light is occluded at each pixel and light up by that amount
1.
Find the fractional shadow coefficients using matting technique
2.
Find ratio of direct to environment light.
Direct
light
Surface
Reflectance
coefficient
Environmental
light
21
content from from Guo et al.
Light Model
Umbra: ti=0
Penumbra = 0 < ti < 1
Ld + Le
Ld
kLd + Le
Le
Relighting :
= (Ldcosθi + Le)Ri
22
Figure from from Guo et al.
Shadow model as a matting problem
Ii = γiFi + (1- γi)Bi
F: Foreground image, B: background Image
Rewrite shadow model as: Ii = ki(LdRi + LeRi) + (1-ki)LeRi
Similar to matting eqn.
Solve for matting:
minimize E(k) = kTL k + λ(k-k’)TD(k-k’)T
optimal k obtained by solving sparse system:
(L + λD)k = λdk’
23
Finding the light ratio
Final r obtained by voting
24
content from from Guo et al.
Results

UCF Dataset (Zhu et al.)

245 images

outdoor scenes, manual annotations
25
content from from Guo et al.
Experiments: Datasets

UCF Dataset (Zhu et al.)

245 images

outdoor scenes, manual annotations
content from from Guo et al.
Experiments: Datasets


108 images, indoor/outdoor, automatic annotation

Evaluate both shadow detection and removal
Input image
Groundtruth
content from from Guo et al.
Results on UCF Dataset
Input image
Groundtruth
Detection
Removal result
content from from Guo et al.
Results on UIUC Dataset
Input image
Detection
Removal result
Groundtruth
content from from Guo et al.
Pixel accuracy
Accuracy
UCF dataset
UIUC dataset
(ours)
Full model
Single region
Zhu et al.
0.900
0.875
0.887
0.883
0.796
-
content from from Guo et al.
Confusion matrices on UCF dataset
full model
Single region classification
0.750
0.250
0.070
0.930
0.515
0.485
0.057
0.947
content from from Guo et al.
Confusion matrices on UCF dataset
full model
Zhu et al. 2010
0.750
0.250
0.639
0.361
0.070
0.930