Report

Descriptive and exploratory statistics Garib Murshudov Contents 1. 2. 3. 4. 5. Itroduction Location Spread Various plots: plots depend on data Histograms, cumulative distributions and all that Purpose of descriptive statistics and various plots • • • • • • maximize insight into a data set uncover underlying structure extract important variables detect outliers and anomalies test underlying assumptions In general: build intuition about the data and problem Descriptive statistics There are two types of the simplest numerical descriptors of a data sets: 1) Values describing location – mean, median, mode 2) Values describing spread – variance, interquartile range Descritpive statistics Histogram of aa Histogram of bb 1000 400 200 0 0 Skewness and kurtosis for normal distribution are zero. 600 Frequency 600 400 0 200 Frequency Kurtosis is a measure of tail: Positive kurtosis: heavier tailthan normal distribution and negative kurtosis:lighter tail than normal distribution 800 Skewness – it is a measure of symmetry of the distribution. Positive skewness: right tail is fatter and negative skewness: left tail is fatter 800 1000 Additional descriptive numerical values: 2 4 6 8 aa Skewness = 1.18 Kurtosis = 2.33 2 4 6 8 bb Skewness = -1.18 Kurtosis = 2.33 10 Location 0.04 0.02 0.00 Density Example: mean = 9.91 median = 9.92 mode = 9.98 0.06 0.08 The simplest information about a data set is about its location. There are three different location parameters: average, median and mode: 1) Average = sum(data)/Ndata 2) Median: proportion of data more than median is equal to that of less than median Histogram of aa 3) Mode: the most occurring data point. -10 0 10 aa 20 30 Location Average is very sensitive to few outliers. If we change one value of data arbitrarily then we can affect average value substantially. However median is not affected very much Example: 13.2 13.2 8.2 10.9 14.3 10.7 8.2 10.9 74.3 10.7 6.6 6.6 9.5 10.8 9.5 10.8 8.8 13.3 8.8 13.3 - Av = 10.63, median = 10.75 - Av = 16.63, median = 10.75 Breakdown point of average is 0, breakdown point of median is 0.5. I.e. you have to change 50% of the data dramatically to affect the median. Median is the most robust estimator Average is the most convenient estimator with nice properties If sample is small then mode it may be impossible to estimate mode. Wikidictionary: The number or proportion of arbitrarily large or small extreme values that must be introduced into a batch or sample to cause the estimator to yield an arbitrarily large result. Simpson’s paradox: batting averages One should be careful in dealing with averages. The most famous paradox related to averages is Simpson’s paradox. Runs Outs Average 1st Ashes MW 270 6 45 SW 500 10 50 2nd Ashes MW 700 10 70 SW 320 4 80 Total MW 970 16 60.25 SW 820 14 58.57 MW – Mark Waugh SW – Steve Waugh Spread 0.3 0.4 Histogram of rn 0.2 Density There are two main indicators of spread of a data set 1) Standard deviation (=(var)1/2). It is a usual indicator of spread. Very easy to calculate. But it is not robust to outliers. One outlier is sufficient to corrupt standard deviation 0.0 0.1 2) Interquartile range - IQR: 50% of the data are within first and third quartile of the data. This indicator is more robust. You need to corrupt 25% of the data to corrupt IQR 0 2 4 6 rn Black vertical lines – quartiles Blue vertical lines - mean+sd, mena-sd Spread: robustness Average is very sensitive to few outliers. If we change one value of data arbitrarily then we can affect average value substantially. However median is not affected very much Example: 13.2 13.2 8.2 10.9 14.3 10.7 8.2 10.9 74.3 10.7 6.6 6.6 9.5 10.8 9.5 10.8 8.8 13.3 8.8 13.3 - sd = 2.45, IQR = 3.65 - sd = 20.37, IQR = 3.65 Breakdown point of sd is 0, breakdown point of median is 0.25. I.e. you have to change at 25% of the data dramatically to affect IQR. IQR is the much more robust than sd Wikidictionary: The number or proportion of arbitrarily large or small extreme values that must be introduced into a batch or sample to cause the estimator to yield an arbitrarily large result. Tukey’s five number summaries One of the important books on statistical data analysis is: Tukey, JW. (1977) Exploratory data analysis After this book there was explosion of exploratory data analysis. I.e. visualisation of datasets and modelling based on visual analysis. One of the suggestions in this book is five number summary of data sets. Essentially these numbers are (although in Tukey’s book different numbers are suggested): Minimum, 1st quartile, median, 3rd quartile, maximum. These numbers are calculated by R with the command summary. For example: A = 13.2 8.2 10.9 14.3 10.7 6.6 summary(A) Min. 1st Qu. Median Mean 6.600 8.975 10.750 10.630 9.5 10.8 8.8 13.3 3rd Qu. 12.620 Max. 14.300 Various plots In general data visualisation is dependent on the type of data and the system it comes from. For some of the data sets it can be suggested to use some general plots. These include: 1) Box and whisker plot – boxplot 2) Histograms 3) Cumulative distribution plots 4) QQ plots 12 10 8 20 18 16 14 12 10 8 Boxplots are convenient ways to visualise one dimensional data. It shows minimum maximum, first quartile, median and third quartile – visual representation of five number summary. This plot can indicate if the distribution of the data is symmetric. This plot may indicate outliers – if one of the points is too different from others – e.g. it is outside the interval (median + 2*IQR) 14 Boxplots Side by side boxplot 0 5 10 15 20 25 Boxplots can be used visual comparison of data, e.g. effects of different treatments. A B C D E Effect of different insecticides F Boxplots Boxplots are just a schematic plots. Sometimes they must mask out some of the features of the data. Classic example is Lord Rayleigh’s data on measurement of densities of nitrogen derived from different sources which lead to the discovery of Argon. Rayleigh was led into the investigation by small anomalies he found in measurements of the density of nitrogen purified by different methods. Those different methods led to different quantities of nitrogen, and thus to different proportions of nitrogen and a hitherto unsuspected atmospheric gas. Argon was the first noble gas isolated. Ramsay's subsequent work isolated helium and discovered neon, krypton, and xenon by the end of the century. Ramsay and Rayleigh were awarded Nobel Prizes in 1904. Rayleigh was awarded the physics prize for argon, while Ramsay was awarded the chemistry prize for argon and the family of noble gases. 2.304 2.302 2.300 2.298 2.300 2.302 2.304 2.306 2.308 2.310 2.298 2.310 2.308 2.306 2.304 Nitrogen 2.302 2.300 2.298 2.308 2.306 2.310 2.308 2.306 2 4 6 8 10 2.304 2.302 2.300 2.298 2.300 2.302 2.304 Index 2.298 Nitrogen 2.310 By looking at the boxplot we do not see any peculiarity in the data. However one can notice that whiskers are very close to the edges of the box, i.e. minimum and maximum are close to first and third quartile respectively. When you see that then you should be suspicious about the data. If we do side by side plot of scatter (dot) plot and boxplot we see peculiar behaviour. There seem to be two classes. Let us use boxplot for different sources of nitrogen. There is definitely two classes. One derived from air and another from other sources. 2.306 2.308 2.310 Boxplots 2 4 6 8 10 Index 14 Air NoAir 14 Insectsprays revisited 25 20 15 0 5 10 15 10 5 0 N Insects 20 25 If we do side by side scatter and boxplot of Insectsparys data we see that there is some peculiarity for spray F. I do not know the reason but it may be interesting to investigate if you see something like that in your data. 0 10 30 Index 50 70 A B C D E F Histograms Histogram of rn1 5 10 rn1 Histogram of rn2 60 0 5 rn1 10 15 0 0 0 0 50 20 40 Frequency 150 Frequency 100 3000 Frequency 1000 2000 6000 4000 2000 Frequency 8000 4000 200 10000 80 Histogram of rn2 0 -5 10 20 30 40 50 60 rn2 0 5 rn1 Scott DW, Multivariate Density Estimation 10 0 10 20 30 rn2 40 0 10 20 30 rn2 Nbin=50 Histogram of rn1 Histogram of rn1 400 500 0 Nbin=500 Nbin=5 300 Frequency 0 0 100 500 200 1000 Frequency 1500 600 2000 Histograms are good way of visualisation of 1D data (there are high dimensional versions also). If there are enough data points then histograms may indicate the potential distribution, multimodality, skewness. For visually pleasing histograms number of bins to calculate histograms is important. Too many bins might be very noisy, too few bins can mask out important features. Histogram of rn2 50 40 50 Cumulative frequency (probability) plot Histograms represent density of probability distribution. To plot histograms we must divide the range of data into bins and then count the number of data points in each bin (for bin number n we need to count the number of data points obeying this: xi ≤ y < xi+1 where xi is the bin boundary and y is the observation). For each bin we may have very small number of data points and therefore their variation may be large resulting in noisy histograms. Cumulative frequency (probability) plots are another way of representing data. In this case we count the number of data points below given point (all y for which y < xi). As we see the number of data points become larger and larger as xi approches to the maximum value of the data points. P Cumulative frequency plots Cumulative distributions may indicate if the data points have normal distribution or heavy tail or some other peculiarities. These plots can also help to select appropriate distribution. However these plots are hard to interpret by their own -3 -2 -1 0 x 1 2 3 0.8 0.6 Fn(x) 0.4 0.2 0.0 0.2 0.4 Fn(x) 0.6 0.8 1.0 ecdf(rn2) 0.0 Data standardisation: y = (x-mean(x))/sd(x) 0.0 0.2 0.4 Fn(x) 0.6 0.8 1.0 One way of comparing two distribution would be plotting them on the same plot. To do this we need at least standardise the data. Even after standardisation the range of the data can be very different. ecdf(rn1) 1.0 ecdf(rn) -2 -1 0 1 x 2 3 -2 0 2 x 4 6 QQ plots Quantile-quantile plots are useful when testing distributions assumptions. These plots could indicate if two data sets are from the same distribution, if yes then they can help to transfer linearly one of them into another one. Mathematically: let us say that X is from the distribution with cumulative distribution function (CDF) – F(x) and Y has the distributions G(y). Then by solving: G(y) = F(x) y = G-1(F(x)) we can find relationship between y and x. As it can be seen random variables can be converted from one to another using QQ plots. For example if x is from exponential distribution – F(x) = 1 – exp(-lambda x) and y is from uniform distribution in the interval (a,b): G(y) = (y-a)/(b-a) then we need to solve: (y-a)/(b-a) = 1-exp(-lambda x) y = b – (b-a) exp(-lambda x), if we see exponential function then we may have this particular relationship. QQ plots Example: uniform and exponential distributions Empirical 0.4 0.6 0.8 0.0 0.0 0.2 0.2 0.4 ru 1 - exp(-3 * xx) 0.6 0.8 1.0 1.0 Theoretical 0.0 0.5 1.0 1.5 r1 2.0 2.5 0.0 0.5 1.0 1.5 2.0 2.5 xx QQ norm QQnorm is the special case of QQ plot – it is a quantile quantile plot against normal distribution. QQ norm can already indicate some properties of the data. 1) Outliers: Normal Too large value Too small value Normal Q-Q Plot Normal Q-Q Plot 2 Sample Quantiles -2 Sample Quantiles -4 0 -6 0 -1 -2 -2 -8 -3 Sample Quantiles 4 1 0 6 2 2 8 3 Normal Q-Q Plot -3 -2 -1 0 1 2 Theoretical Quantiles 3 -3 -2 -1 0 1 2 Theoretical Quantiles 3 -3 -2 -1 0 1 2 Theoretical Quantiles 3 QQ norm 2) Bimodality, skewness (note that small curviture for small and large values can be expected) Normal Skewed to left Bimodal Normal Q-Q Plot Normal Q-Q Plot 3 3 Normal Q-Q Plot Convex 0 2 1 Sample Quantiles 1 Sample Quantiles 0 0 -1 -1 -1 -2 -3 Sample Quantiles 3 1 4 2 2 5 Curviture -3 -2 -1 0 1 2 Theoretical Quantiles 3 -3 -2 -1 0 1 2 Theoretical Quantiles 3 -3 -2 -1 0 1 2 Theoretical Quantiles 3 QQ norm 3) Heavier tail t distribution, df = 3 Normal Histogram of rtt Normal Q-Q Plot Heavier tails 0 Sample Quantiles 40 -2 60 Frequency 100 0 -6 20 -4 50 0 Frequency 80 2 150 4 100 6 120 200 Histogram of rn -3 -2 -1 0 rn 1 2 3 -6 -4 -2 0 rtt 2 4 6 -3 -2 -1 0 1 2 Theoretical Quantiles 3 QQ norm If distribution of two random variables have the same form then we may derive linear transformation of data vs another one. -30 -20 -10 0 rn1 10 20 30 Qqplot: one data vs another. Slope and intercept of the line gives linear transformation needed: y = a + bx and a = 3, b=10 -3 -2 -1 0 rn 1 2 3 Conclusions • Average and variance are usual measures for location and spread of the data. However they are not robust. Median and IQR are more rbust • Boxplot is good way of summarising data, however it might mask out features of the data • QQ plot can be used to check distribution assumptions References • Tukey, JW. Exploratory data analysis • Scott DW, Multivariate Density Estimation