hide long namesshow long names
hide short namesshow short names
Integer type:  int32  int64  nag_int  show int32  show int32  show int64  show int64  show nag_int  show nag_int

PDF version (NAG web site, 64-bit version, 64-bit version)
Chapter Contents
Chapter Introduction
NAG Toolbox

NAG Toolbox Chapter Introduction

G01 — simple calculations on statistical data

Scope of the Chapter

This chapter covers three topics:

Background to the Problems

Plots, Descriptive Statistics and Exploratory Data Analysis

Plots and simple descriptive statistics are generally used for one of two purposes:
Exploratory data analysis (EDA) is used to pick out the important features of the data in order to guide the choice of appropriate models. EDA makes use of simple displays and summary statistics. These may suggest models or transformations of the data which can then be confirmed by further plots. The process is interactive between you, the data, and the program producing the EDA displays.
The summary statistics consist of two groups. The first group are those based on moments; for example mean, standard deviation, coefficient of skewness, and coefficient of kurtosis (sometimes called the ‘excess of kurtosis’, which has the value 0 for the Normal distribution). These statistics may be sensitive to extreme observations and some robust versions are available in Chapter G07. The second group of summary statistics are based on the order statistics, where the ith order statistic in a sample is the ith smallest observation in that sample. Examples of such statistics are minimum, maximum, median, hinges and quantiles.
In addition to summarising the data by using suitable statistics the data can be displayed using tables and diagrams. Such data displays include frequency tables, stem and leaf displays, box and whisker plots, histograms and scatter plots.

Statistical Distribution Functions and Their Inverses

Statistical distributions are commonly used in three problems:
Random variables can be either discrete (i.e., they can take only a limited number of values) or continuous (i.e., can take any value in a given range). However, for a large sample from a discrete distribution an approximation by a continuous distribution, usually the Normal distribution, can be used. Distributions commonly used as a model for discrete random variables are the binomial, hypergeometric, and Poisson distributions. The binomial distribution arises when there is a fixed probability of a selected outcome as in sampling with replacement, the hypergeometric distribution is used in sampling from a finite population without replacement, and the Poisson distribution is often used to model counts.
Distributions commonly used as a model for continuous random variables are the Normal, gamma, and beta distributions. The Normal is a symmetric distribution whereas the gamma is skewed and only appropriate for non-negative values. The beta is for variables in the range 0,1 and may take many different shapes. For circular data, the ‘equivalent’ to the Normal distribution is the von Mises distribution. The assumption of the Normal distribution leads to procedures for testing and interval estimation based on the χ2, F (variance ratio), and Student's t-distributions.
In the hypothesis testing situation, a statistic X with known distribution under the null hypothesis is evaluated, and the probability α of observing such a value or one more ‘extreme’ value is found. This probability (the significance) is usually then compared with a preassigned value (the significance level of the test), to decide whether the null hypothesis can be rejected in favour of an alternate hypothesis on the basis of the sample values. Many tests make use of those distributions derived from the Normal distribution as listed above, but for some tests specific distributions such as the Studentized range distribution and the distribution of the Durbin–Watson test have been derived. Nonparametric tests as given in Chapter G08, such as the Kolmogorov–Smirnov test, often use statistics with distributions specific to the test. The probability that the null hypothesis will be rejected when the simple alternate hypothesis is true (the power of the test) can be found from the noncentral distribution.
The confidence interval problem requires the inverse calculation. In other words, given a probability α, the value x is to be found, such that the probability that a value not exceeding x is observed is equal to α. A confidence interval of size 1-2α, for the quantity of interest, can then be computed as a function of x and the sample values.
The required statistics for either testing hypotheses or constructing confidence intervals can be computed with the aid of functions in this chapter, and Chapter G02 (for regression), Chapter G04 (for analysis of designed experiments), Chapter G13 (for time series), and Chapter E04 (for nonlinear least squares problems).
Pseudorandom numbers from many statistical distributions can be generated by functions in Chapter G05.

Testing for Normality and Other Distributions

Methods of checking that observations (or residuals from a model) come from a specified distribution, for example, the Normal distribution, are often based on order statistics. Graphical methods include the use of probability plots. These can be either P-P plots (probability–probability plots), in which the empirical probabilities are plotted against the theoretical probabilities for the distribution, or Q-Q plots (quantile–quantile plots), in which the sample points are plotted against the theoretical quantiles. Q-Q plots are more common, partly because they are invariant to differences in scale and location. In either case if the observations come from the specified distribution then the plotted points should roughly lie on a straight line.
If yi is the ith smallest observation from a sample of size n (i.e., the ith order statistic) then in a Q-Q plot for a distribution with cumulative distribution function F, the value yi is plotted against xi, where Fxi=i-α/n-2α+1, a common value of α being 12 . For the Normal distribution, the Q-Q plot is known as a Normal probability plot.
The values xi used in Q-Q plots can be regarded as approximations to the expected values of the order statistics. For a sample from a Normal distribution the expected values of the order statistics are known as Normal scores and for an exponential distribution they are known as Savage scores.
An alternative approach to probability plots are the more formal tests. A test for Normality is the Shapiro and Wilk's W Test, which uses Normal scores. Other tests are the χ2 goodness-of-fit test and the Kolmogorov–Smirnov test; both can be found in Chapter G08.

Distribution of Quadratic Forms

Many test statistics for Normally distributed data lead to quadratic forms in Normal variables. If X is a n-dimensional Normal variable with mean μ and variance-covariance matrix Σ then for an n by n matrix A the quadratic form is
Q=XTAX.  
The distribution of Q depends on the relationship between A and Σ: if AΣ is idempotent then the distribution of Q will be central or noncentral χ2 depending on whether μ is zero.
The distribution of other statistics may be derived as the distribution of linear combinations of quadratic forms, for example the Durbin–Watson test statistic, or as ratios of quadratic forms. In some cases rather than the distribution of these functions of quadratic forms the values of the moments may be all that is required.

Energy Loss Distributions

An application of distributions in the field of high-energy physics where there is a requirement to model fluctuations in energy loss experienced by a particle passing through a layer of material. Three models are commonly used:
(i) Gaussian (Normal) distribution;
(ii) the Landau distribution;
(iii) the Vavilov distribution.
Both the Landau and the Vavilov density functions can be defined in terms of a complex integral. The Vavilov distribution is the more general energy loss distribution with the Landau and Gaussian being suitable when the Vavilov parameter κ is less than 0.01 and greater than 10.0 respectively.

Vectorized Functions

A number of vectorized functions are included in this chapter. Unlike their scalar counterparts, which take a single set of parameters and perform a single function evaluation, these functions take vectors of parameters and perform multiple function evaluations in a single call. The input arrays to these vectorized functions are designed to allow maximum flexibility in the supply of the parameters by reusing, in a cyclic manner, elements of any arrays that are shorter than the number of functions to be evaluated, where the total number of functions evaluated is the size of the largest array.
To illustrate this we will consider nag_stat_prob_gamma_vector (g01sf), a vectorized version of nag_stat_prob_gamma (g01ef), which calculates the probabilities for a gamma distribution. The gamma distribution has two parameters α and β therefore nag_stat_prob_gamma_vector (g01sf) has four input arrays, one indicating the tail required (tail), one giving the value of the gamma variate, g, whose probability is required (g), one for α (a) and one for β (b). The lengths of these arrays are ltail, lg, la and lb respectively.
For sake of argument, lets assume that ltail=1, lg=2, la=3 and lb=4, then maxltail,lg,la,lb=4 values will be returned. These four probabilities would be calculated using the following parameters:
i Tail g α β
1 tail1 g1 a1 b1
2 tail1 g2 a2 b2
3 tail1 g1 a3 b3
4 tail1 g2 a1 b4

Recommendations on Choice and Use of Available Functions

Descriptive statistics / Exploratory analysis, 
    plots, 
        box and whisker nag_stat_plot_box_whisker (g01as)
        histogram nag_stat_plot_histogram (g01aj)
        Normal probability (Q − Q) plot nag_stat_plot_scatter_normal (g01ah)
        scatter plot nag_stat_plot_scatter_2var (g01ag)
        stem and leaf nag_stat_plot_stem_leaf (g01ar)
    summaries, 
        frequency / contingency table, 
            one variable nag_stat_frequency_table (g01ae)
            two variables, with  χ 2 and Fisher's exact test nag_stat_contingency_table (g01af)
        mean, variance, skewness, kurtosis (one variable), 
            combine summaries nag_stat_summary_onevar_combine (g01au)
            from frequency table nag_stat_summary_freq (g01ad)
            from raw data nag_stat_summary_onevar (g01at)
        mean, variance, sums of squares and products (two variables) nag_stat_summary_2var (g01ab)
        median, hinges / quartiles, minimum, maximum nag_stat_5pt_summary (g01al)
        quantiles, 
            approximate, 
                large  data stream of fixed size nag_stat_quantiles_stream_fixed (g01an)
                large data stream of unknown size nag_stat_quantiles_stream_arbitrary (g01ap)
            unordered vector nag_stat_quantiles (g01am)
        rolling window, 
            mean, standard deviation (one variable) nag_stat_moving_average (g01wa)
Distributions, 
    Beta, 
        central, 
            deviates, 
                scalar nag_stat_inv_cdf_beta (g01fe)
                vectorized nag_stat_inv_cdf_beta_vector (g01te)
            probabilities and probability density function, 
                scalar nag_stat_prob_beta (g01ee)
                vectorized nag_stat_prob_beta_vector (g01se)
        non-central, 
            probabilities nag_stat_prob_beta_noncentral (g01ge)
    binomial, 
        distribution function, 
            scalar nag_stat_prob_binomial (g01bj)
            vectorized nag_stat_prob_binomial_vector (g01sj)
    Dickey–Fuller unit root test, 
        probabilities, nag_stat_prob_dickey_fuller_unit (g01ew)
    Durbin–Watson statistic, 
        probabilities nag_stat_prob_durbin_watson (g01ep)
    energy loss distributions, 
        Landau, 
            density nag_stat_pdf_landau (g01mt)
            derivative of density nag_stat_pdf_landau_deriv (g01rt)
            distribution nag_stat_prob_landau (g01et)
            first moment nag_stat_pdf_landau_moment1 (g01pt)
            inverse distribution nag_stat_inv_cdf_landau (g01ft)
            second moment nag_stat_pdf_landau_moment2 (g01qt)
        Vavilov, 
            density nag_stat_pdf_vavilov (g01mu)
            distribution nag_stat_prob_vavilov (g01eu)
            initialization nag_stat_init_vavilov (g01zu)
    F: 
        central, 
            deviates, 
                scalar nag_stat_inv_cdf_f (g01fd)
                vectorized nag_stat_inv_cdf_f_vector (g01td)
            probabilities, 
                scalar nag_stat_prob_f (g01ed)
                vectorized nag_stat_prob_f_vector (g01sd)
        non-central, 
            probabilities nag_stat_prob_f_noncentral (g01gd)
    gamma, 
        deviates, 
            scalar nag_stat_inv_cdf_gamma (g01ff)
            vectorized nag_stat_inv_cdf_gamma_vector (g01tf)
        probabilities, 
            scalar nag_stat_prob_gamma (g01ef)
            vectorized nag_stat_prob_gamma_vector (g01sf)
        probability density function, 
            scalar nag_stat_pdf_gamma (g01kf)
            vectorized nag_stat_pdf_gamma_vector (g01kk)
    Hypergeometric, 
        distribution function, 
            scalar nag_stat_prob_hypergeom (g01bl)
            vectorized nag_stat_prob_hypergeom_vector (g01sl)
    Kolomogorov–Smirnov, 
        probabilities, 
            one-sample nag_stat_prob_kolmogorov1 (g01ey)
            two-sample nag_stat_prob_kolmogorov2 (g01ez)
    Normal, 
        bivariate, 
            probabilities nag_stat_prob_bivariate_normal (g01ha)
        multivariate, 
            probabilities nag_stat_prob_multi_normal (g01hb)
            probability density function, 
                vectorized nag_stat_pdf_multi_normal_vector (g01lb)
            quadratic forms, 
                cumulants and moments nag_stat_moments_quad_form (g01na)
                moments of ratios nag_stat_moments_ratio_quad_forms (g01nb)
        univariate, 
            deviates, 
                scalar nag_stat_inv_cdf_normal (g01fa)
                vectorized nag_stat_inv_cdf_normal_vector (g01ta)
            probabilities, 
                scalar nag_stat_prob_normal (g01ea)
                vectorized nag_stat_prob_normal_vector (g01sa)
            probability density function, 
                scalar nag_stat_pdf_normal (g01ka)
                vectorized nag_stat_pdf_normal_vector (g01kq)
            reciprocal of Mill's Ratio nag_stat_mills_ratio (g01mb)
            Shapiro and Wilk's test for Normality nag_stat_test_shapiro_wilk (g01dd)
    Poisson, 
        distribution function, 
            scalar nag_stat_prob_poisson (g01bk)
            vectorized nag_stat_prob_poisson_vector (g01sk)
    Student's t: 
        central, 
            bivariate, 
                probabilities nag_stat_prob_bivariate_students_t (g01hc)
            multivariate, 
                probabilities nag_multi_students_t (g01hd)
            univariate, 
                deviates, 
                    scalar nag_stat_inv_cdf_students_t (g01fb)
                    vectorized nag_stat_inv_cdf_students_t_vector (g01tb)
                probabilities, 
                    scalar nag_stat_prob_students_t (g01eb)
                    vectorized nag_stat_prob_students_t_vector (g01sb)
        non-central, 
            probabilities nag_stat_prob_students_t_noncentral (g01gb)
    Studentized range statistic, 
        deviates nag_stat_inv_cdf_studentized_range (g01fm)
        probabilities nag_stat_prob_studentized_range (g01em)
    von Mises, 
        probabilities nag_stat_prob_vonmises (g01er)
     χ 2: 
        central, 
            deviates nag_stat_inv_cdf_chisq (g01fc)
            probabilities nag_stat_prob_chisq (g01ec)
            probability of linear combination nag_stat_prob_chisq_lincomb (g01jd)
        non-central, 
            probabilities nag_stat_prob_chisq_noncentral (g01gc)
            probability of linear combination nag_stat_prob_chisq_noncentral_lincomb (g01jc)
        vectorized deviates nag_stat_inv_cdf_chisq_vector (g01tc)
        vectorized probabilities nag_stat_prob_chisq_vector (g01sc)
Scores, 
    Normal scores, 
        accurate nag_stat_normal_scores_exact (g01da)
        approximate nag_stat_normal_scores_approx (g01db)
        variance-covariance matrix nag_stat_normal_scores_var (g01dc)
    Normal scores, ranks or exponential (Savage) scores nag_stat_ranks_and_scores (g01dh)
Note:  the Student's t, χ2, and F functions do not aim to achieve a high degree of accuracy, only about four or five significant figures, but this should be quite sufficient for hypothesis testing. However, both the Student's t and the F-distributions can be transformed to a beta distribution and the χ2-distribution can be transformed to a gamma distribution, so a higher accuracy can be obtained by calls to the gamma or beta functions.
Note:  nag_stat_ranks_and_scores (g01dh) computes either ranks, approximations to the Normal scores, Normal, or Savage scores for a given sample. nag_stat_ranks_and_scores (g01dh) also gives you control over how it handles tied observations. nag_stat_normal_scores_exact (g01da) computes the Normal scores for a given sample size to a requested accuracy; the scores are returned in ascending order. nag_stat_normal_scores_exact (g01da) can be used if either high accuracy is required or if Normal scores are required for many samples of the same size, in which case you will have to sort the data or scores.

Working with Streamed or Extremely Large Datasets

The majority of the functions in this chapter are ‘in-core’, that is all the data required must be held in memory prior to calling the function. In some situations this might not be possible, for example, when working with extremely large datasets or where all of the data is not available at once (i.e., the data is being streamed).
There are five functions in this chapter applicable to datasets of this form:
nag_stat_summary_onevar (g01at) computes the mean, variance and the coefficients of skewness and kurtosis for a single variable.
nag_stat_summary_onevar_combine (g01au), takes the results from two calls to nag_stat_summary_onevar (g01at) and combines them, returning the mean, variance and the coefficients of skewness and kurtosis for the combined dataset. This function allows the easy utilization of more than one processor to spread the computational burden inherent in summarising a very large dataset.
nag_stat_quantiles_stream_fixed (g01an) and nag_stat_quantiles_stream_arbitrary (g01ap) compute the approximate quantiles for a dataset of known and unknown size respectively.
nag_stat_moving_average (g01wa) computes the mean and standard deviation in a rolling window.
In addition, see nag_correg_ssqmat (g02bu) and nag_correg_ssqmat_combine (g02bz) for functions to summarise two or more variables.

References

Hastings N A J and Peacock J B (1975) Statistical Distributions Butterworth
Kendall M G and Stuart A (1969) The Advanced Theory of Statistics (Volume 1) (3rd Edition) Griffin
Tukey J W (1977) Exploratory Data Analysis Addison–Wesley

PDF version (NAG web site, 64-bit version, 64-bit version)
Chapter Contents
Chapter Introduction
NAG Toolbox

© The Numerical Algorithms Group Ltd, Oxford, UK. 2009–2015