Report

Noa Privman Horesh December 2012 Many uses for fusion: Visualization – fuse between bands Sharpening - fusion between hyperspectaral image and panchromatic image Detection and classification - fusion between hyperspectaral image and panchromatic/ FOPEN SAR / LIDAR image Visualization In lecture 3 – Displaying of Hyperspectral Images on RGB Displays we saw several algorithms: 1BT Based Band Selection Principal Components Analysis (PCA) Spectral Weighting Envelopes Visualization – cont’ Another method for getting better visualization - hierarchical fusion based on vector quantization and bilateral filtering. Hierarchical Fusion Using Vector Quantization for Visualization of Hyperspectral Images A typical hyperspectral image data set in remote sensing contains a few hundred images to be fused into a single image (for grayscale) or three images (for RGB). Fusing all the bands together requires heavy calculation and a lot of memory Visualization - Hierarchical Fusion For the hyperspectral image cube of dimensions (X*Y *N), vector quantization (V Q) based fusion is applied across a contiguous subset of dimensions (X*Y*P) to generate B = N/P different fused images at the first stage of hierarchy In the subsequent levels of hierarchy, contiguous images are grouped together in a smaller subset and fused using ’bilateral filtering’. Visualization - Hierarchical Fusion – cont’ Images I1 to IN from N contiguous bands are organized into Group1 to GroupB, using uniform grouping. So each group has P = N/B images each of size X*Y . First stage - Each group is individually fused using Vector quantization Fused using Vector quantization Vector Quantization used to compress the information and manipulate the data in a way that maintain the most important features. Each image Ik is divided into sub-blocks of size mxm giving rise to (XxY )/m2 image blocks. In a given group there are IVn = (XxYxP)/(m2) image sub- blocks. Generate first code-vector Convert these image vectors to one dimensional vectors each of size m2 and generate a cluster (matrix) S of size IVnx m2 The first code-vector (CV (1,1)) of the codebook size 1, can be computed by finding the column wise average of the entire cluster as follows: Generate code book The code-vector (CV (1,1)) is then split into two code-vectors by adding and subtracting a tolerance ε, in order to double the codebook size: Generate code book – cont’ The original cluster S is divided into two clusters – S1 and S2 based on the distortion D1(2,1)and D1(2,2) with respect to the codevectors Comparing the values D1(2,1) (k) and D1(2,2)(k) the image vectors of the cluster S is grouped into two sub-clusters S1 and S2 such that and Generate code book – cont’ The quality of the code-book is enhanced by updating existing code-vectors through calculating the mean of the image-vectors in each sub-cluster S1 and S2. The code-vectors are updated to the new code-vectors. The corresponding distortions are calculated for the complete image vector set S to get updated sub-clusters S1 and S2. Generate code book – cont’ The update repeated until the vector sum of the distortion in the current level is significantly less than the distortion in the previous level Now we have n code-vectors, each of size m2, in the code-book (size nxm2). Fused using Vector quantization Each image Ii rearrange to a matrix of size (XxY/m2, m2). The rearranged image is now compared with all the n code-vectors with respect to MSE The MSE values of all the P images for a given sub-block position with all the codevectors are then added. The code-vector CVi that gives the minimum sum of MSE values is selected as the ith sub-block of the fused image IF Hierarchical Fusion - vector quantization At the end of first stage fusion, there are B fused images (I1,1 to I1,B) which are the input images for second level of hierarchy. Fusion using Bilateral Filtering bilateral filtering is used only from the second hierarchical level following the redundancy removal in the first stage through Vector quantization. A bilateral filter A bilateral filter is an edge-preserving and noise reducing smoothing filter. The intensity value at each pixel in an image is replaced by a weighted average of intensity values from nearby pixels. This weight is based on a Gaussian distribution. This preserves sharp edges by systematically looping through each pixel and according weights to the adjacent pixels accordingly. Fusion using Bilateral Filtering Compute the bilateral filtered image: Calculate the weight at each pixel (x, y) for each image: Fusion using Bilateral Filtering The fused Image of the hyperspectral cube subset IF is given by The 1st and the 81st image of the urban image cube (Palo Alto) from Hyperion dataset Results (a) (b) (c) Sharpening Combine the high spatial and the high spectral resolutions in order to obtain the complete and accurate description of the observed scene. The following method will be describe: Unmixing-based constrained nonnegative matrix factorization (UCNMF) Unmixing-based constrained nonnegative matrix factorization Nonnegative matrix factorization (NMF) for hyperspectral unmixing The hyperspectral data is a 3D-array V ∈ RL×K store the original hyperspectral data. V = WH + N W ∈ RL×S - the spectral signature matrix H ∈ RS×K is the abundance matrix Nonnegative matrix factorization (NMF) for hyperspectral unmixing – cont’ To unmix the hyperspectral data, NMF could be conducted: Euc(W, H) 1 2 V WH 2 V 2 1 i WH ij j Minimize the square of the Euclidean distance between V and WH ij 2 Unmixing-based constrained nonnegative matrix factorization (UCNMF) for image fusion After creating the abundance matrix, the weighted fusion method is adopted. min F(W, H) 1 V WH 2 2 s .t ., W 0 , H 0 Therefore, we have the fused data Vf : Vf = W(αH + (1 − α)P) Preserve the spectral information of the original hyperspectral image The fuse image does not hold the same spectral quality as the original hyperspectral image causing spectral distortion . Constraint function: V V k S f i 1 2 fi 2 V i V , fi V i Which is equivalent to: 2 tr V V . * V V tr V V . * V V SV T f f T f T f T Final Fusion model min J(W, H) = F(W, H) + βS(Vf ) s.t. W ≥ 0, H ≥ 0 Vf = W(αH + (1 − α)P) This is an optimization problem Algorithm (Outline: Lin-PG for UCNMF). Given 0 < ı < 1, 0 < < 1, 0 < ε < 1. Set γ0 = 1. Initialize the matrices W ≥ 0, H ≥ 0. Calculate the J W 1, H 1 2. For k = 1, 2, . . . (a) Assign γk ← γk-1 . (b) If γk satisfies (1), repeatedly increase it by γk←γk/δ until either γk does not satisfy (1) or W, H keep the same before and after the change of γk Else repeatedly decrease γk by γk←γk· δ until γk satisfies (1). (c) Update W by (2), H by (3). (d) Calculate the P J W k , H k 3. Repeat step 2, until satisfying the stopping condition given in (4). 4. Obtaining the fused image Vf = W(αH + (1 − α)P) 1. F F 1 2 3 4 Results proposed method UCNMF has the advantage that it could advance the spatial resolution of the hyperspectral image without losing much its color information Detection and classification Fusing data from hyperspectral imaging (HSI) sensors with data from other sensors can enhance overall detection and classification performance. Fusing HSI data with foliage-penetration synthetic aperture radar (FOPEN SAR) data - feature level Fusing HSI data with high-resolution imaging (HRI) data - data and feature level HSI and FOPEN SAR Data Fusion FOPEN SAR and HSI sensors detection capabilities complement each other. FOPEN SAR typically operates at 20 to 700 MHz. It penetrates foliage and detects targets under tree canopy, but has significant clutter returns from trees. HSI is capable of subpixel detection and material identification HSI and FOPEN SAR Data Fusion Both SAR and HSI systems may suffer substantial false-alarm and missed detection rates because of their respective background clutter. Reduction in spectral dimensionality to the HSI data cube in order to extract the spectral features HSI and FOPEN SAR Data Fusion PCA is used to decorrelate data and maximize the information content in a reduced number of features A matched-filtering algorithm with thresholding was then applied to the HSI data to detect all pixels of fabric nets. HSI fabric-net detection with a matchedfiltering algorithm (left) and terrain classification map (right). The map shows background classes for roads, grass, trees, and shadow regions; these classes result from an unsupervised data-clustering operation that uses the first five principal components Combined FOPEN SAR-HSI Analysis and Fusion The SAR data processed with pixel grouping and threshold. Combined analyses, retained only SAR detections from either open areas or around fabric nets indicated in the HSI data. SAR detections that corresponded to identifications of trees, far-from-open areas, or nets on the HSI were considered false alarms. SAR detection confirmed using HSI material identified There are several strong SAR detections on the left side of the open area. Three pixels match well with military gray-tan paint, indicating the presence of a vehicle, possibly military; This match confirms the SAR detection. HSI and HRI Data Fusion Sharpening the HSI data conduct a combined spatial-spectral analysis Background classification and anomaly detection are first obtained from HSI data. Applying the results to the sharpened HSI data provides enhanced background classification and target detection. HSI and HRI Data Fusion – cont’ The HRI data provide target and background boundaries with spatial edge detection. These edges, combined with results from the sharpened HSI data, spatially enhance the definition of targets and backgrounds. Finally, spectral-matched filtering for target detection is applied to the sharpened HSI data. References Shah, P.; Jayalakshmi, M.; Merchant, S.N.; Desai, U.B.; , "Hierarchical fusion using vector quantization for visualization of hyperspectral images," Information Fusion (FUSION), 2011 Proceedings of the 14th International Conference on , vol., no., pp.1-8, 5-8 July 2011 Z. Zhang, et al., Hyperspectral and panchromatic image fusion using unmixing-based constrained nonnegative matrix factorization, Optik - Int. J. Light Electron Opt. (2012), http://dx.doi.org/10.1016/j.ijleo.2012.04.022 Multisensor Fusion with Hyperspectral Imaging Data: Detection and Classification Su May Hsu , Hsiao-hua K. Burke