Report

Multivariate data and multivariate analysis Multivariate data Multivariate data - values recorded for several random variables on a number of units. Unit 1 Variable 1 … Variable q … x11 . . . . . . n xn1 x1q . . . … . . . xnq n – number of units q – number of variables recorded on each unit xij - value of the jth variable for the ith unit Multivariate analysis Multivariate statistical analysis – simultaneous statistical analysis of a collection of variables (e.g.handle apples, oranges, and pears at the same time). Simultaneous statistical analysis of a collection of variables improves upon separate univariate analyses of each variable by using information about the relationships between the variables. The general aim of most multivariate analysis is to uncover, display, or extract any “signal” in the data in the presence of noise and to discover what the data have to tell us. Multivariate analysis - Exploratory methods allow the detection of possible unanticipated patterns in the data, opening up a wide range of competing explanations. These methods are characterized by an emphasis on the importance of graphical display and visualization of the data and the lack of any associated probabilistic model that would allow for formal inferences (discussed in this course). - Statistical inferences methods are used when individuals from a multivariate data set have been sampled from some population and the investigator wishes to test a well defined hypothesis about the parameters of that population’s probability density function (multivariate normal). History of the development of multivariate analysis - late 19th century Francis Galton and Karl Pearson quantified the relationship between offspring and parental characteristics and developed the correlation coefficient. - early 20th century Charles Spearman introduced the factor analysis while investigating correlated intelligence quotient (IQ) tests. Over the next two decades, Spearman’s work was extended by Hotelling and by Thurstone. - in 1920s Fisher’s introduction of the analysis of variance was followed by its multivariate generalization, multivariate analysis of variance, based on work by Bartlett and Roy. History of the development of multivariate analysis - At the beginning, the computational aids to take the burden of the vast amounts of arithmetic involved in the application of the multivariate methods were very limited. - In the early years of the 21st century, the wide availability of cheap and extremely powerful personal computers and laptops, and flexible statistical software has meant that all the methods of multivariate analysis can be applied routinely even to very large data sets. - Application of multivariate techniques to large data sets was named “data mining” (the nontrivial extraction of implicit, previously unknown and potentially useful information from data). multivariate data ExampleExample of ofmultivariate data individual sex age IQ depression health weight 1 Male 21 120 Yes Very good 150 2 Male 43 NA No Very good 160 3 Male 22 135 No Average 135 4 Male 86 150 No Very poor 140 5 Male 60 92 Yes Good 110 6 Female 16 130 Yes Good 110 7 Female NA 150 Yes Very good 120 8 Female 43 NA Yes Average 120 9 Female 22 84 No Average 105 10 Female 80 70 No Good 100 NA – Not Available Number of units n=10 Number of variables q=7 Example of multivariate data Types of measurements Nominal - Unordered categorical variables (e.g. treatment allocation, the sex of the responder, hair color). Ordinal - Where there is an ordering but no implication of equal distance between the different points of the scale (e.g. educational level: no schooling, primary, secondary, or tertiary education). Interval - Where there are equal differences between successive points on the scale but the position of zero is arbitrary (e.g. measurement of temperature using the Celsius or Fahrenheit scales). Ratio – The highest level of measurement, relative magnitudes of scores as well as the differences between them. The position of zero is fixed (e.g. absolute measure of temperature (Kelvin), age, weight, and length). Example of multivariate data Missing values Observations and measurements that should have been recorded but for one reason or another, were not (e.g. non-response in sample surveys, dropouts in longitudinal data). How to deal with missing values? 1. complete - case analysis by omitting any case with a missing value on any of the variables (not recommended because might lead to misleading conclusion and inferences). 2. available-case analysis – exploit incomplete information by using all the cases available to estimate the quantities of interest (difficulties arise when missing data is not missing completely at random). 3. multiple imputation – a Monte Carlo technique in which the missing values are replaced by m>1 simulated versions (3< m <10) (the most appropriate way). Covariance Example of multivariate data Covariance of two random variables is a measure of their linear dependence. Cov( Xi, Xj ) E ( Xi i )( Xj j ) i E ( Xi ) j E ( Xj ) E - expectation If the two variables are independent of each other, their covariance is equal to zero. Larger values of the covariance show greater degree of linear dependence between two variables. Example of multivariate data Covariance If i j the covariance of the variable with itself is simply its variance. The variance of variable Xi is: E((Xi i) ) 2 2 In a multivariate data with q observed variables, there are q variances and q(q-1)/2 covariances. Covariance depends on the scales on which the two variables are measured. Example of multivariate d Covariance matrix 12 12 21 22 q1 q 2 1q 2q q2 ij covariance of Xi and Xj i2 - varianceof variableXi ij ji Example of multivariate Measure datad Measurements of chest, waist, and hips on a sample 20 men and women . R data “Measure” chest 34 37 38 36 38 43 40 38 40 41 waist 30 32 30 33 29 32 33 30 30 32 hips 32 37 36 39 33 38 42 40 37 39 gender male male male male male male male male male male chest 36 36 34 33 36 37 34 36 38 35 waist 24 25 24 22 26 26 25 26 28 23 hips 35 37 37 34 37 37 38 37 40 35 gender female female female female female female female female female female Example of multivariate d Covariance Calculate the covariance matrix for the numerical variables : chest, waist, hips in “Measure” data. We remove the categorical variable “gender” (column 4). > cov(Measure[,1:3] chest waist chest 6.631579 6.368421 waist 6.368421 12.526316 hips 3.052632 3.684211 hips 3.052632 3.684211 5.894737 Example of multivariate d Covariance Covariance matrix for gender female in “Measure” data >cov(subset(Measure,Measure$gender=="female")[,c(1:3)]) chest waist hips chest 2.277778 2.166667 1.500000 waist 2.166667 2.988889 2.633333 hips 1.500000 2.633333 2.900000 Covariance matrix for gender male in “Measure” data >cov(subset(Measure, Measure$gender=="male")[,c(1:3)]) chest waist hips chest 6.7222222 0.9444444 3.944444 waist 0.9444444 2.1000000 3.077778 hips 3.9444444 3.0777778 9.344444 Example of multivariate d Correlation Correlation is independent of the scales of the two variables. Correlation coefficient ( ij ) is the covariance divided by the product of standard deviations of the two variables. ij ij ij i where 2 i i - standard deviation The correlation coefficient lies between -1 and +1and gives a measure of the linear relationship of the variables Xi and Xj. It is positive if high values of Xi are associated with high values of Xj and negative if high values of Xi are associated with low values of Xj. Example of multivariate d Correlation Correlation matrix for “Measure” data > cor(Measure[,1:3]) chest waist hips chest 1.0000000 0.6987336 0.4882404 waist 0.6987336 1.0000000 0.4287465 hips 0.4882404 0.4287465 1.0000000 Example of multivariate d Distances Distance between the units in the data is often of considerable importance in some multivariate techniques. The most common measure used : Euclidian distance dij q 2 ( x x ) ik jk k 1 xik and xjk, k = 1,…, q variable values for units i and j. When the variables in a multivariate data set are on different scales we need to do a standardization before calculating the distances (e.g. divide each variable by its standard deviation). Example of multivariate d Distances The distance matrix for the first 12 observations in “Measure” data after standardization. >dist(scale(Measure[,1:3],center = FALSE)) 1 2 3 4 5 6 7 8 9 10 11 2 0.17 3 0.15 0.08 4 0.22 0.07 0.14 5 0.11 0.15 0.09 0.22 6 0.29 0.16 0.16 0.19 0.21 7 0.32 0.16 0.20 0.13 0.28 0.14 8 0.23 0.11 0.11 0.12 0.19 0.16 0.13 9 0.21 0.10 0.06 0.16 0.12 0.11 0.17 0.09 10 0.27 0.12 0.13 0.14 0.20 0.06 0.09 0.11 0.09 11 0.23 0.28 0.22 0.33 0.19 0.34 0.38 0.25 0.24 0.32 12 0.22 0.24 0.18 0.28 0.18 0.30 0.32 0.20 0.20 0.28 0.06 of multivariate d MultivariateExample normal density function Multivariate normal density function for two variables x1 and x2: f ( x1 , x2 ); ( 1 , 2 ), 1 , 2 , ) 2 x 2 1 x x x 2 1 / 2 2 2 (21 2 (1 )) exp 1 1 2 1 1 2 2 2 1 2 2 2(1 ) 1 µ1 and µ2 – population means of the two variables σ1 and σ2 - population variances ρ - population correlation between two variables X1 and X2 Linear combinations of the variables are themselves normally distributed. Example of multivariate d Multivariate normal density function Methods to test the multivariate normal distribution: - normal probability plots for each variable separately - convert each multivariate observation to a single number before plotting (i.e. each q-dimensional observation xi could be converted into a generalized distance di2 , giving a measure of the distance of the particular observation from the mean vector of the complete sample x̅ ). di2 = (xi - x̅)T S-1 (xi - x̅) S- sample covariance matrix If the observations are from a multivariate normal distribution, then distances have approximately a chi-squared distribution with q degrees 2 of freedom and are denoted by the symbol q . Example of multivariate d Air pollution data Air pollution in 41 cities in the USA. R data “USairpollution” Variables: SO2: SO2 content of air in micrograms per cubic meter temp: average annual temperature in degrees Fahrenheit manu: number of manufacturing enterprises employing 20 or more workers popul: population size (1970 census) in thousands wind: average annual wind speed in miles per hour precip: average annual precipitation in inches predays: average number of days with precipitation per year Multivariate normal density function Read the “USairpollution” data with the first column as row names: >USairpollution=read.csv("E:/Multivariate_analysis/Data/USairpollution.csv" ,header=T,row.names=1) Normal probability plots for “manu” and “popul” variables in “USairpollution” data. > qqnorm(USairpollution$manu,main=“manu”) > qqline(USairpollution$manu) > qqnorm(USairpollution$popul,main=“popul”) > qqline(USairpollution$popul) Multivariate normal density function popul 2500 0 1000 Sample Quantiles 2500 1000 0 Sample Quantiles manu -2 -1 0 1 2 Theoretical Quantiles -2 -1 0 1 2 Theoretical Quantiles Multivariate normal density function Normal probability plots for each variable separately in “USairpollution” data. layout(matrix(1:8,nc=2)) sapply(colnames(USairpollution), function(x){ qqnorm(USairpollution[[x]], main=x) qqline(USairpollution[[x]]) }) 0 1 2 11 wind 6 8 Sample Quantiles 20 60 -1 -2 -1 0 1 2 precip 60 45 -2 -1 0 1 2 10 30 50 temp Sample Quantiles Theoretical Quantiles 75 Theoretical Quantiles -2 -1 0 1 2 manu predays -2 -1 0 1 2 0 1500 3500 popul -2 -1 0 1 2 Theoretical Quantiles 40 100 Theoretical Quantiles Sample Quantiles Theoretical Quantiles 0 1500 Sample Quantiles Sample Quantiles Sample Quantiles -2 Theoretical Quantiles Sample Quantiles The plots for SO2 concentration and precipitation both deviate considerably from linearity, and the plots for manufacturing and population show evidence of a number of outliers. SO2 -2 -1 0 1 2 Theoretical Quantiles Multivariate normal density function Chi-square plot: >x=USairpollution >cm=colMeans(x) >S=cov(x) >d=apply(x,1,function(x)t(x-cm)%*%solve(S)%*%(x-cm)) >plot(qc <- qchisq((1:nrow(x)-1/2)/nrow(x),df=6), sd<-sort(d), xlab=expression(paste(chi[6]^2,"Quantile")), ylab="Ordered distances",xlim=range(qc)*c(1,1.1)) >oups=which(rank(abs(qc-sd),ties="random")>nrow(x)-3) >text(qc[oups],sd[oups]-1.5,names(oups)) >abline(a=0,b=1) Multivariate normal density function Plotting the ordered distances against the corresponding quantiles of the appropriate chi-square distribution should lead to a straight line through the origins. Chi-square plot is also useful for detecting outliers (i.g. Chicago, Phoenix, Providence). 25 Chicago 15 Phoenix Providence 5 Ordered distances Chi-square plot 5 10 2 Quantile 6 15