Report

ACCELERATING SPARSE CANONICAL CORRELATION ANALYSIS FOR LARGE BRAIN IMAGING GENETICS DATA Jingw en Yan, Hui Zhang, Lei Du, Eric Wernert, Andew J. Saykin, Li Shen OUTLINE • Imaging Genetics • Sparse Canonical Correlation Analysis (SCCA) • Computational Challenges and Methods • Data Simulation • Experimental Results IMAGING GENETICS Genes Cells Systems UCI, S. Potkin et al. Behavior: Disorders, Complex interactions, phenomena, diseases. IMAGING GENETICS Underlying Biological Pathway and Mechanism IMAGING GENETICS Candidate Gene/SNP Biological Pathway Genome-wide Single ROI Risacher et al 2010 Sloan et al 2010 Potkin et al 2009; Saykin et al 2010 Circuit Risacher et al 2013 AV45 ROIs & APOE Swaminathan et al 2012 PiB ROIs & amyloid pathway Potkin et al 2009 Mol Psych schizophrenia study Ho et al 2010 FTO; Reiman et al PNAS 2009 Chiang et al 2012 SNP/Gene networks & WM integrity Shen et al 2010 ROIs; Stein et al 2010 voxels Whole Brain OUTLINE • Imaging Genetics • Sparse Canonical Correlation Analysis (SCCA) • Computational Challenges and Methods • Data Simulation • Experimental Results SCCA X1 Y1 X1 Y1 X1 Y1 X2 Y2 X2 Y2 X2 W’X Y2 Xu Yv X3 Y3 X3 Y3 X3 Y3 Xn Yn Xn Yn Xn Yn Massive Univariate Analysis Multivariate Multiple Regression Canonical Correlation Analysis SCCA • Sparse canonical correlation analysis (SCCA) • R package: Penalized Multivariate Analysis (PMA) (Witten, et al, 2009) max, subject to • • • • • = 1, = 1 1 ≤ 1 , 2 ≤ 2 X, Y : imaging and genetics data respectively 1 , 2 : sparse penalties, mostly 1 norm For simplicity, assuming = and = Bi-convex and non differentiable problem Iterative solution SCCA • Sparse canonical correlation analysis (SCCA) • Problem max, subject to • = 1, = 1, ≤ 1 , 1 Iterative solution 1. ← arg max , subject to = 1, 2. ← arg max , subject to = 1, • 1 S( , ∆) S( , ∆) 2 1 1 ≤ 1 ≤ 2 ← , ( , ∆) is the soft thresholding operator and ∆ ≥ 0 is chosen so that u 1 ≤ c1 ≤ 2 OUTLINE • Imaging Genetics • Sparse Canonical Correlation Analysis (SCCA) • Computational Challenges and Methods • Data Simulation • Experimental Results COMPUTATIONAL CHALLENGES • Example SCCA run at a small scale • • • • • • Scale up • • • • • Participants: 1000 Genotype: 3,200 SNPs Phenotype: 10,000 voxels Permutation: 10,000 permutation tests Running time: more than 12,000 hours Genotype (array): 6M SNPs Genotype (NGS): 40M variants Phenotype: 200K voxels, imaging, cognitive and biomarker Permutation: 10M permutation to reach p=10-7 Parameter tuning via cross-validation • • 10-fold cross-validation coupled with an 11-by-11 grid search SCCA runs: 10×11×11 = 1,210 ACCELERATION WITH MKL • Intel Math Kernel Library (MKL) • • • MKL has been optimized to utilize • • • • accelerate application performance and reduce development time highly vectorized and threaded linear algebra, fast fourier transforms (FFT), vector math and statistics functions multiple processing cores wider vector units more varied architectures available in a high end system MKL can provide parallelism transparently and speed up programs with supported math routines without changing code. • Compiling R with MKL ACCELERATION WITH OFFLOAD MODEL • Xeon Phi SE10P Coprocessor • 60 cores with 8GB GDDR5 • Intel x86 instruction set • Usage of familiar programming models, software, and tools • Pros • The host system can offload computing workload partially to the Xeon Phi • Independently run a compatible program COMPUTATIONAL PLATFORM • Texas Advanced Computing Center Stampede cluster • • Each computing node • • • • • MKL + offload Two Intel Xeon E5-2680 processors each with eight cores @2.7GHz. 32GB DDR3 memory The Xeon Phi SE10P Coprocessor has 61 cores with 8GB GDDR5 The NVIDIA K20 GPUs on each node have 5GB of on-board GDDR5 Software • • CentOS 6.3. Stock R 3.01 package compiled with the Intel compilers (v.13) and built with MKL v.11. OUTLINE • Imaging Genetics • Sparse Canonical Correlation Analysis (SCCA) • Computational Challenges and Methods • Data Simulation • Experimental Results SYNTHETIC DATA (GENETICS) • FREGENE genome simulator • Simulate sequence-like data over large genomic regions in large diploid populations • Simulated data • N=1,000 diploid individuals over 20,000 generations • 10 Mb genome with the average mutation rate as 2.5e-8 /site/generation • 3,274 SNPs with minor allele frequency (MAF) greater than 0.05 included • Four SNP data sets (i.e., g500, g1000, g2000, and g3274) by taking the first 500, 1,000, 2,000, and 3,274 SNPs from the entire data, respectively. SYNTHETIC DATA (GENETICS) SYNTHETIC DATA (IMAGING) • Assumption • Each image with multiple regions of interest (ROIs) • Voxel within each ROI highly correlated • Simulation • Random positive definite non-overlapping group structured covariance matrix • Apply Cholesky decomposition to obtain the background imaging data • Individual: N=1000, Size: 100x100 • We created three sets of phenotypic imaging data (i.e., p1000, p5000, and p10000), consisting of 1,000, 5,000 and 10,000 voxels respectively SYNTHETIC DATA (IMAGING) OUTLINE • Imaging Genetics • Sparse Canonical Correlation Analysis (SCCA) • Computational Challenges and Methods • Data Simulation • Experimental Results RESULTS • R snowfall package (sfLapply) with MKL and offload model Baseline Parallel (MKL+ offload) RESULTS Correlation coefficient between the first pair of canonical components • Accelerated SCCA implementations yielded the same results • These correlation coefficients are close to the ground truth value of 1 RESULTS CONCLUSION • Initial steps to accelerate the SCCA implementation for brain imaging genetics applications. • Parallelism achieved in system implementation level to accelerate linear algebra computation using math kernel library (MKL) and partial offloading computing workload. • The 2-fold speedup, although encouraging, is still insufficient to handle extremely large-scale neuroimaging genetics data • • millions of image voxels and millions of SNPs. Future work • Big data analytic strategies at the parallel computing model level • • Parallelization of multiplicative algorithms using MapReduce and CUDA. Application to accelerate enhanced SCCA models as well as other bimultivariate statistical models for analyzing brain imaging genetics data. ACKNOWLEDGEMENT This research was supported by • • • • • • NIH R01 LM011360 NIH U01 AG024904 NIH RC2 AG036535 NIH R01 AG19771 NIH P30 AG10133 NSF IIS-1117335 Thank you