Report

Active Inference for Retrieval in Camera Networks Daozheng Chen1, Mustafa Bilgic2, Lise Getoor1, David Jacobs1, Lilyana Mihalkova1, Tom Yeh1 1 Department of Computer Science, University of Maryland, College Park 2 Department of Computer Science, Illinois Institute of Technology Problem • Search camera network videos to retrieve frames containing specified individuals. query Time query Time Related Work • Person re-identification [Wang et al. ’07] • Graphical Models for camera networks [Loy et al. ’09] • Tracking over camera networks [Song et al. ’07] • Active Learning [Settles ’09] Our Contributions • Map video frames in a camera network onto a graphical model and use a collective classification algorithm to predict frame states and perform frame retrieval. • Apply active inference to direct human attention to portions of the videos which are most likely to have the biggest performance improvement. Graphical Structures Collective Classification Active Inference Active Inference Outline • • • • • Graphical model construction Iterative classification algorithm Active inference Experiment Conclusion Graphical Model Construction • Temporal neighbors (TN) • Frames from the previous and next k time steps within the same camera. • Positively correlated spatial neighbors (PSN) • Correlation of the labels of two camera is greater than some threshold. • Negatively correlated spatial neighbors (NSN) • Correlation of the labels of two camera is less than some threshold. Temporal Neighbors (k = 1) Positively correlated spatial neighbors Negatively correlated spatial neighbors Graphical Structures Outline • • • • • Graphical model construction Iterative Classification Algorithm Active Inference Experiment Conclusion The Iterative Classification Algorithm (ICA) • Local Models (LM). • The label of a frame is only dependent on its features. • Relational Models (RM). • The label of a frame is dependent on its features and its neighbors’ current labels • First apply the local model for initialization, and then use the relational model iteratively until predicted labels converge. [Sen et al. ’08] The Iterative Classification Algorithm (ICA) • Local Models (LM). • Logistic regression as the classifier. • Cosine similarity based on signatures using bag-of-feature model as features Fq = [fq1,fq2,…,fqn] F = [f1,f2,…,fn] COS(Fq,F) The Iterative Classification Algorithm (ICA) • Relational Models (RM). • Logistic regression as the classifier. • Use aggregation function to construct a feature vector encoding neighbors’ information. F = [f21,f22,…,f2n] The Iterative Classification Algorithm (ICA) • Relational Models (RM). • Logistic regression as the classifier. • Use aggregation function to construct a feature vector encoding neighbors’ information. F = [f21,f22,…,f2n] FTN = [fTN1,fTN2] The Iterative Classification Algorithm (ICA) • Relational Models (RM). • Logistic regression as the classifier. • Use aggregation function to construct a feature vector encoding neighbors’ information. F = [f21,f22,…,f2n] FTN = [fTN1,fTN2] FPSN = [fPSN1,fPSN2] The Iterative Classification Algorithm (ICA) • Relational Models (RM). • Logistic regression as the classifier. • Use aggregation function to construct a feature vector encoding neighbors’ information. F = [f21,f22,…,f2n] FTN = [fTN1,fTN2] FPSN = [fPSN1,fPSN2] FNSN = [fNSN1,fNSN2] The Iterative Classification Algorithm (ICA) • Relational Models (RM). • Logistic regression as the classifier. • Use aggregation function to construct a feature vector encoding neighbors’ information. F = [f21,f22,…,f2n] FTN = [fTN1,fTN2] FPSN = [fPSN1,fPSN2] FNSN = [fNSN1,fNSN2] FRM Outline • • • • • Graphical model construction Iterative Classification Algorithm Active Inference Experiment Conclusion Active Inference • The retrieval algorithm can request the correct labels for some frames at inference time. [Rattigan et al. ’07] • Subsequent inference using ICA is based on these corrected labels. • Common methods for selecting frames to label: • Random (RND). • Uniform (UNI). • Most certain to be relevant (MR). • Most uncertain (UNC) • Reflect and Correct. [Bilgic and Getoor. ’09] Reflect and Correct (RAC) [Bilgic and Getoor. ’TKDD09] Adaptive RAC (MLI) We adapt RAC with the following the 10 features to train a classifier to determine whether a frame, f, is misclassified – Most Likely Incorrect (MLI) Entropy(f) Entropy of the probability estimate of f based on RM Entropy(TN(f)) Average entropy of the probability estimate of TN of f based on RM Entropy(PSN(f)) Average entropy of the probability estimate PSN of f based on RM Entropy(NSN(f)) Average entropy of the probability estimate of NSN of f based on RM KL(f) KL divergence between probability estimates of LM and RM of f KL(TN(f)) KL divergence between probability estimates of LM and RM of TN of f KL(PSN(f)) KL divergence between probability estimates of LM and RM of PSN of f KL(NSN(f)) KL divergence between probability estimates of LM and RM of NSN of f IsRelevant(f) Whether is f is predicted to be relevant by RM %Agree(f) the percentage of TN and PSN whose predicted or corrected labels agree with the predicted label of f. Outline • • • • • Graphical model construction Iterative Classification Algorithm Active Inference Experimental Evaluation Conclusion Dataset [Ding et al. ’10] Queries Region of Interests • Background subtraction to determine region of interest in the frame. • Densely sample key points in the regions • Use color histogram in RGB space to describe the region spanned by a key point • Quantized the descriptor according to learned 500 code words. • Produce a single signature for a video frame. Spatial Topology Methods for Comparison • Active inference based on LM using RND, UNI, MR, UNC, MLI • Active inference based on RM using RND, UNI, MR, UNC, MLI • Average accuracy and Average 11-average precision as measurement Results • UNC-LM has the best performance when results are based on LM. • RM always perform better than LM does under the same sampling method. • UNC-RM and MLI always perform better. • MLI never perform worse than MLI does. Outline • • • • • Graphical model construction Iterative Classification Algorithm Active Inference Experiment Conclusion Conclusion • Using a graphical model provides significant performance improvements in frame retrieval. • A simple method that captures the frame uncertainty has an advantage over other baseline methods. • Our adaptation of RAC has overall better performance. Questions?