From Interactive to Semantic Image Segmentation Varun Gulshan Supervisors: Prof. Andrew Blake Prof. Andrew Zisserman 20 Jan 2012 Two segmentation tasks sky building tree tree background person object car bench Interactive segmentation car road Semantic segmentation Thesis Flow Chapter 3: Chapter 4: Superpixel based classification Fully automatic segmentation Texture Features Star convexity Segmenting humans Low level cues Mid level cues Bounding box interaction + Top down cues Interactive segmentation Chapter 6: Chapter 5: Semantic segmentation Chapter 3: Features for interactive segmentation Low level texture features for improving interactive segmentation methods. Texture features Pure Texture Feature (L-shape): Texture + Gray Feature (L-shape): Texture features Pure Texture Feature (Plus-shape): Texture + Gray Feature (Plus-shape): Camouflage Image Dataset Introduced a dataset of 50 Camouflage images, to demonstrate the power of texture features. RGB Gray Quantitative evaluation +14% +21% +4% Huge gain in accuracy obtained using texture features on top of gray scale images. Significant improvement on top of RGB images. +7% Chapter 4: Star Convexity and Extensions Mid level shape constraints for reducing user effort in interactive segmentation systems. Chapter 4: Star convexity Single Star Multiple Stars Geodesic Star Robot user evaluation False negative False positive Initial brush strokes New Brush Stroke New brush stroke placed Segmentation output with current interaction Error segmentation New Brush Stroke Centre of connected component Biggest connected component Process is repeated upto 20 strokes Updated segmentation Segmentation after 20 strokes Robot user evaluation Method SP-IG SP-SIG SP-LIG BJ RW GSCseq Effort 17.78 15.77 15.14 12.35 12.31 9.63 Our method takes least effort Chapter 5: Learning to segment humans Using top down cues to segment specific object categories. Segmenting humans Bounding box (given/detected) Top down HOG prediction Bottom up refinement Kinect Data Acquisition RGB image Kinect scene labels Cleaned up Ground truth Dataset of roughly 3500 images acquired using the Kinect Top down learning Local Image Local HOG Local mask Classifier trained to predict segmentation masks for local windows based on their HOG descriptor. Bottom up refinement ….. Top down segmentation Local Color model window Local color model unaries Final segmentation Chapter 6: Semantic segmentation Fully automatic segmentation based upon learning from multiple superpixelisations. Combing multiple superpixelisations GlobalPb Veksler Various methods to learn from multiple superpixelisations explored: 1. Avg-Indep 2. Avg-Union 3. LPβ-Indep 4. IofR-Joint QuickShift Multiple superpixelisations Single superpixelisation Quantitative evaluation +7% +5% +6% Combining multiple superpixelisations improves performance. +3% Novel pairwise features .... .... CRF trained jointly for appearance and novel pairwise features. Over to you!