NAG Library Function Document

nag_regsn_ridge_opt (g02kac)

nag_regsn_ridge_opt (Nag_OrderType order, Integer n, Integer m, const double x[], Integer pdx, const Integer isx[], Integer ip, double tau, const double y[], double *h, Nag_PredictError opt, Integer *niter, double tol, double *nep, Nag_EstimatesOption orig, double b[], double vif[], double res[], double *rss, Integer *df, Nag_OptionLOO optloo, double perr[], NagError *fail)

3 Description

A linear model has the form:

y = c + X β + ε,

where

$y$ is an $n$ by $1$ matrix of values of a dependent variable;
$c$ is a scalar intercept term;
$X$ is an $n$ by $m$ matrix of values of independent variables;
$β$ is an $m$ by $1$ matrix of unknown values of parameters;
$ε$ is an $n$ by $1$ matrix of unknown random errors such that variance of $ε = σ^{2} I$ .

Let

\tilde{X}

be the mean-centred

X

and

\tilde{y}

the mean-centred

y

. Furthermore,

\tilde{X}

is scaled such that the diagonal elements of the cross product matrix

{\tilde{X}}^{T} \tilde{X}

are one. The linear model now takes the form:

\tilde{y} = \tilde{X} \tilde{β} + ε .

Ridge regression estimates the parameters

\tilde{β}

in a penalised least squares sense by finding the

\tilde{b}

that minimizes

{‖\tilde{X} \tilde{b} - \tilde{y}‖}^{2} + h {‖\tilde{b}‖}^{2}, h > 0,

where

‖\cdot‖

denotes the

ℓ_{2}

-norm and

h

is a scalar regularisation or ridge parameter. For a given value of

h

, the parameter estimates

\tilde{b}

are found by evaluating

\tilde{b} = {({\tilde{X}}^{T} \tilde{X} + h I)}^{- 1} {\tilde{X}}^{T} \tilde{y} .

Note that if

h = 0

the ridge regression solution is equivalent to the ordinary least squares solution.

Rather than calculate the inverse of (

{\tilde{X}}^{T} \tilde{X} + h I

) directly, nag_regsn_ridge_opt (g02kac) uses the singular value decomposition (SVD) of

\tilde{X}

. After decomposing

\tilde{X}

into

U D V^{T}

where

U

and

V

are orthogonal matrices and

D

is a diagonal matrix, the parameter estimates become

\tilde{b} = V {(D^{T} D + h I)}^{- 1} D U^{T} \tilde{y} .

A consequence of introducing the ridge parameter is that the effective number of parameters,

γ

, in the model is given by the sum of diagonal elements of

D^{T} D {(D^{T} D + h I)}^{- 1},

see Moody (1992) for details.

Any multi-collinearity in the design matrix

X

may be highlighted by calculating the variance inflation factors for the fitted model. The

j

th variance inflation factor,

v_{j}

, is a scaled version of the multiple correlation coefficient between independent variable

j

and the other independent variables,

R_{j}

, and is given by

v_{j} = \frac{1}{1 - R_{j}}, j = 1, 2, \dots, m .

The

m

variance inflation factors are calculated as the diagonal elements of the matrix:

{({\tilde{X}}^{T} \tilde{X} + h I)}^{- 1} {\tilde{X}}^{T} \tilde{X} {({\tilde{X}}^{T} \tilde{X} + h I)}^{- 1},

which, using the SVD of

\tilde{X}

, is equivalent to the diagonal elements of the matrix:

V {(D^{T} D + h I)}^{- 1} D^{T} D {(D^{T} D + h I)}^{- 1} V^{T} .

Although parameter estimates

\tilde{b}

are calculated by using

\tilde{X}

, it is usual to report the parameter estimates

b

associated with

X

. These are calculated from

\tilde{b}

, and the means and scalings of

X

. Optionally, either

\tilde{b}

b

may be calculated.

The method can adopt one of four criteria to minimize while calculating a suitable value for

h

(a)

Generalised cross-validation (GCV):

\frac{n s}{{(n - γ)}^{2}};

(b)

Unbiased estimate of variance (UEV):

\frac{s}{n - γ};

(c)

Future prediction error (FPE):

\frac{1}{n} (s + \frac{2 γ s}{n - γ});

(d)

Bayesian information criterion (BIC):

\frac{1}{n} (s + \frac{\log (n) γ s}{n - γ});

where

s

is the sum of squares of residuals. However, the function returns all four of the above prediction errors regardless of the one selected to minimize the ridge parameter,

h

. Furthermore, the function will optionally return the leave-one-out cross-validation (LOOCV) prediction error.

4 References

Hastie T, Tibshirani R and Friedman J (2003) The Elements of Statistical Learning: Data Mining, Inference and Prediction Springer Series in Statistics

Moody J.E. (1992) The effective number of parameters: An analysis of generalisation and regularisation in nonlinear learning systems In: Neural Information Processing Systems (eds J E Moody, S J Hanson, and R P Lippmann) 4 847–854 Morgan Kaufmann San Mateo CA

5 Arguments

1: order – Nag_OrderTypeInput

On entry: the order argument specifies the two-dimensional storage scheme being used, i.e., row-major ordering or column-major ordering. C language defined storage is specified by

order = Nag_RowMajor

. See Section 3.2.1.3 in the Essential Introduction for a more detailed explanation of the use of this argument.

Constraint:

order = Nag_RowMajor

or Nag_ColMajor.

2: n – IntegerInput

On entry:

n

, the number of observations.

Constraint:

n > 1

3: m – IntegerInput

On entry: the number of independent variables available in the data matrix

X

Constraint:

m \leq n

4: x[ $\dim$ ] – const doubleInput

Note: the dimension, dim, of the array x must be at least

$\max (1, pdx \times m)$ when $order = Nag_ColMajor$ ;
$\max (1, n \times pdx)$ when $order = Nag_RowMajor$ .

The

(i, j)

th element of the matrix

X

is stored in

$x [(j - 1) \times pdx + i - 1]$ when $order = Nag_ColMajor$ ;
$x [(i - 1) \times pdx + j - 1]$ when $order = Nag_RowMajor$ .

On entry: the values of independent variables in the data matrix

X

5: pdx – IntegerInput

On entry: the stride separating row or column elements (depending on the value of order) in the array x.

Constraints:

if $order = Nag_ColMajor$ , $pdx \geq n$ ;
if $order = Nag_RowMajor$ , $pdx \geq m$ .

6: isx[m] – const IntegerInput

On entry: indicates which

m

independent variables are included in the model.

$isx [j - 1] = 1$: The $j$ th variable in x will be included in the model.
$isx [j - 1] = 0$: Variable $j$ is excluded.

Constraint:

isx [j - 1] = 0 ​ or ​ 1

, for

j = 1, 2, \dots, m

7: ip – IntegerInput

On entry:

m

, the number of independent variables in the model.

Constraints:

$1 \leq ip \leq m$ ;
Exactly ip elements of isx must be equal to $1$ .

8: tau – doubleInput

On entry: singular values less than tau of the SVD of the data matrix

X

will be set equal to zero.

Suggested value:

tau = 0.0

Constraint:

tau \geq 0.0

9: y[n] – const doubleInput

On entry: the

n

values of the dependent variable

y

10: h – double *Input/Output

On entry: an initial value for the ridge regression parameter

h

; used as a starting point for the optimization.

Constraint:

h > 0.0

On exit: h is the optimized value of the ridge regression parameter

h

11: opt – Nag_PredictErrorInput

On entry: the measure of prediction error used to optimize the ridge regression parameter

h

. The value of opt must be set equal to one of:

$opt = Nag_GCV$: Generalised cross-validation (GCV);
$opt = Nag_UEV$: Unbiased estimate of variance (UEV)
$opt = Nag_FPE$: Future prediction error (FPE)
$opt = Nag_BIC$: Bayesian information criteron (BIC).

Constraint:

opt = Nag_GCV

Nag_UEV

Nag_FPE

Nag_BIC

12: niter – Integer *Input/Output

On entry: the maximum number of iterations allowed to optimize the ridge regression parameter

h

Constraint:

niter \geq 1

On exit: the number of iterations used to optimize the ridge regression parameter

h

within tol.

13: tol – doubleInput

On entry: iterations of the ridge regression parameter

h

will halt when consecutive values of

h

lie within tol.

Constraint:

tol > 0.0

14: nep – double *Output

On exit: the number of effective parameters,

γ

, in the model.

15: orig – Nag_EstimatesOptionInput

On entry: if

orig = Nag_EstimatesOrig

, the parameter estimates

b

are calculated for the original data; otherwise

orig = Nag_EstimatesStand

and the parameter estimates

\tilde{b}

are calculated for the standardized data.

Constraint:

orig = Nag_EstimatesOrig

Nag_EstimatesStand

16: b[ $ip + 1$ ] – doubleOutput

On exit: contains the intercept and parameter estimates for the fitted ridge regression model in the order indicated by isx. The first element of b contains the estimate for the intercept;

b [j]

contains the parameter estimate for the

j

th independent variable in the model, for

j = 1, 2, \dots, ip

17: vif[ip] – doubleOutput

On exit: the variance inflation factors in the order indicated by isx. For the

j

th independent variable in the model,

vif [j - 1]

is the value of

v_{j}

, for

j = 1, 2, \dots, ip

18: res[n] – doubleOutput

On exit:

res [i - 1]

is the value of the

i

th residual for the fitted ridge regression model, for

i = 1, 2, \dots, n

19: rss – double *Output

On exit: the sum of squares of residual values.

20: df – Integer *Output

On exit: the degrees of freedom for the residual sum of squares rss.

21: optloo – Nag_OptionLOOInput

On entry: if

optloo = Nag_WantLOO

, the leave-one-out cross-validation estimate of prediction error is calculated; otherwise no such estimate is calculated and

optloo = Nag_NoLOO

Constraint:

optloo = Nag_NoLOO

Nag_WantLOO

22: perr[ $5$ ] – doubleOutput

On exit: the first four elements contain, in this order, the measures of prediction error: GCV, UEV, FPE and BIC.

optloo = Nag_WantLOO

perr [4]

is the LOOCV estimate of prediction error; otherwise

perr [4]

is not referenced.

23: fail – NagError *Input/Output

The NAG error argument (see Section 3.6 in the Essential Introduction).

6 Error Indicators and Warnings

NE_2_INT_ARG_CONS: On entry, $ip = 〈value〉$ ; $m = 〈value〉$ .
Constraint: $1 \leq ip \leq m$ .
NE_ALLOC_FAIL: Dynamic memory allocation failed.
NE_BAD_PARAM: On entry, argument $〈value〉$ had an illegal value.
NE_INT: On entry, $n = 〈value〉$ .
Constraint: $n > 1$ .; On entry, $niter = 〈value〉$ .
Constraint: $niter \geq 1$ .; On entry, $pdx = 〈value〉$ .
Constraint: $pdx > 0$ .
NE_INT_2: On entry, $m = 〈value〉$ and $n = 〈value〉$ .
Constraint: $m \leq n$ .; On entry, $pdx = 〈value〉$ ; $n = 〈value〉$ .
Constraint: $pdx \geq n$ .; On entry, $pdx = 〈value〉$ and $m = 〈value〉$ .
Constraint: $pdx \geq m$ .
NE_INT_ARG_CONS: On entry, $ip = 〈value〉$ .
Constraint: $sum (isx) = ip$ .
NE_INT_ARRAY_VAL_1_OR_2: On entry, $isx [〈value〉] = 〈value〉$ .
Constraint: $isx [j - 1] = 0$ or $1$ .
NE_INTERNAL_ERROR: An internal error has occurred in this function. Check the function call and any array sizes. If the call is correct then please contact NAG for assistance.
NE_REAL: On entry, $h = 〈value〉$ .
Constraint: $h > 0.0$ .; On entry, $tau = 〈value〉$ .
Constraint: $tau \geq 0.0$ .; On entry, $tol = 〈value〉$ .
Constraint: $tol > 0.0$ .
NE_SVD_FAIL: SVD failed to converge.
NW_TOO_MANY_ITER: Maximum number of iterations used.

7 Accuracy

Not applicable.

8 Further Comments

nag_regsn_ridge_opt (g02kac) allocates internally

\max (5 \times (n - 1), 2 \times ip \times ip) + (n + 3) \times ip + n

elements of double precision storage.

9 Example

This example reads in data from an experiment to model body fat, and a ridge regression is calculated that optimizes GCV prediction error.

NAG Library Function Documentnag_regsn_ridge_opt (g02kac)

+− Contents

1 Purpose

2 Specification

3 Description

4 References

5 Arguments

6 Error Indicators and Warnings

7 Accuracy

8 Further Comments

9 Example

9.1 Program Text

9.2 Program Data

9.3 Program Results

NAG Library Function Document

nag_regsn_ridge_opt (g02kac)