g02 Chapter Contents
g02 Chapter Introduction
NAG Library Manual

NAG Library Function Documentnag_regsn_ridge (g02kbc)

1  Purpose

nag_regsn_ridge (g02kbc) calculates a ridge regression, with ridge parameters supplied by you.

2  Specification

 #include #include
 void nag_regsn_ridge (Nag_OrderType order, Integer n, Integer m, const double x[], Integer pdx, const Integer isx[], Integer ip, const double y[], Integer lh, const double h[], double nep[], Nag_ParaOption wantb, double b[], Integer pdb, Nag_VIFOption wantvf, double vf[], Integer pdvf, Integer lpec, const Nag_PredictError pec[], double pe[], Integer pdpe, NagError *fail)

3  Description

A linear model has the form:
 $y = c+Xβ+ε ,$
where
• $y$ is an $n$ by $1$ matrix of values of a dependent variable;
• $c$ is a scalar intercept term;
• $X$ is an $n$ by $m$ matrix of values of independent variables;
• $\beta$ is a $m$ by $1$ matrix of unknown values of parameters;
• $\epsilon$ is an $n$ by $1$ matrix of unknown random errors such that variance of ${\epsilon =\sigma }^{2}I$.
Let $\stackrel{~}{X}$ be the mean-centred $X$ and $\stackrel{~}{y}$ the mean-centred $y$. Furthermore, $\stackrel{~}{X}$ is scaled such that the diagonal elements of the cross product matrix ${\stackrel{~}{X}}^{\mathrm{T}}\stackrel{~}{X}$ are one. The linear model now takes the form:
 $y~ = X~ β~ + ε .$
Ridge regression estimates the parameters $\stackrel{~}{\beta }$ in a penalised least squares sense by finding the $\stackrel{~}{b}$ that minimizes
 $X~ b~ - y~ 2 + h b~ 2 , h>0 ,$
where $‖·‖$ denotes the ${\ell }_{2}$-norm and $h$ is a scalar regularization or ridge parameter. For a given value of $h$, the parameters estimates $\stackrel{~}{b}$ are found by evaluating
 $b~ = X~T X~+hI -1 X~T y~ .$
Note that if $h=0$ the ridge regression solution is equivalent to the ordinary least squares solution.
Rather than calculate the inverse of (${\stackrel{~}{X}}^{\mathrm{T}}\stackrel{~}{X}+hI$) directly, nag_regsn_ridge (g02kbc) uses the singular value decomposition (SVD) of $\stackrel{~}{X}$. After decomposing $\stackrel{~}{X}$ into $UD{V}^{\mathrm{T}}$ where $U$ and $V$ are orthogonal matrices and $D$ is a diagonal matrix, the parameter estimates become
 $b~ = V DTD+hI -1 DUT y~ .$
A consequence of introducing the ridge parameter is that the effective number of parameters, $\gamma$, in the model is given by the sum of diagonal elements of
 $DT D DT D+hI -1 ,$
see Moody (1992) for details.
Any multi-collinearity in the design matrix $X$ may be highlighted by calculating the variance inflation factors for the fitted model. The $j$th variance inflation factor, ${v}_{j}$, is a scaled version of the multiple correlation coefficient between independent variable $j$ and the other independent variables, ${R}_{j}$, and is given by
 $vj = 1 1-Rj , j=1,2,…,m .$
The $m$ variance inflation factors are calculated as the diagonal elements of the matrix:
 $X~T X~+hI -1 X~T X~ X~T X~+hI-1 ,$
which, using the SVD of $\stackrel{~}{X}$, is equivalent to the diagonal elements of the matrix:
 $V DT D+hI -1 DT D DT D+hI -1 VT .$
Given a value of $h$, any or all of the following prediction criteria are available:
(a) Generalized cross-validation (GCV):
 $ns n-γ 2 ;$
(b) Unbiased estimate of variance (UEV):
 $s n-γ ;$
(c) Future prediction error (FPE):
 $1n s+ 2γs n-γ ;$
(d) Bayesian information criterion (BIC):
 $1n s+ lognγs n-γ ;$
(e) Leave-one-out cross-validation (LOOCV),
where $s$ is the sum of squares of residuals.
Although parameter estimates $\stackrel{~}{b}$ are calculated by using $\stackrel{~}{X}$, it is usual to report the parameter estimates $b$ associated with $X$. These are calculated from $\stackrel{~}{b}$, and the means and scalings of $X$. Optionally, either $\stackrel{~}{b}$ or $b$ may be calculated.

4  References

Hastie T, Tibshirani R and Friedman J (2003) The Elements of Statistical Learning: Data Mining, Inference and Prediction Springer Series in Statistics
Moody J.E. (1992) The effective number of parameters: An analysis of generalisation and regularisation in nonlinear learning systems In: Neural Information Processing Systems (eds J E Moody, S J Hanson, and R P Lippmann) 4 847–854 Morgan Kaufmann San Mateo CA

5  Arguments

1:     orderNag_OrderTypeInput
On entry: the order argument specifies the two-dimensional storage scheme being used, i.e., row-major ordering or column-major ordering. C language defined storage is specified by ${\mathbf{order}}=\mathrm{Nag_RowMajor}$. See Section 3.2.1.3 in the Essential Introduction for a more detailed explanation of the use of this argument.
Constraint: ${\mathbf{order}}=\mathrm{Nag_RowMajor}$ or $\mathrm{Nag_ColMajor}$.
2:     nIntegerInput
On entry: $n$, the number of observations.
Constraint: ${\mathbf{n}}\ge 1$.
3:     mIntegerInput
On entry: the number of independent variables available in the data matrix $X$.
Constraint: ${\mathbf{m}}\le {\mathbf{n}}$.
4:     x[$\mathit{dim}$]const doubleInput
Note: the dimension, dim, of the array x must be at least
• $\mathrm{max}\phantom{\rule{0.125em}{0ex}}\left(1,{\mathbf{pdx}}×{\mathbf{m}}\right)$ when ${\mathbf{order}}=\mathrm{Nag_ColMajor}$;
• $\mathrm{max}\phantom{\rule{0.125em}{0ex}}\left(1,{\mathbf{n}}×{\mathbf{pdx}}\right)$ when ${\mathbf{order}}=\mathrm{Nag_RowMajor}$.
The $\left(i,j\right)$th element of the matrix $X$ is stored in
• ${\mathbf{x}}\left[\left(j-1\right)×{\mathbf{pdx}}+i-1\right]$ when ${\mathbf{order}}=\mathrm{Nag_ColMajor}$;
• ${\mathbf{x}}\left[\left(i-1\right)×{\mathbf{pdx}}+j-1\right]$ when ${\mathbf{order}}=\mathrm{Nag_RowMajor}$.
On entry: the values of independent variables in the data matrix $X$.
5:     pdxIntegerInput
On entry: the stride separating row or column elements (depending on the value of order) in the array x.
Constraints:
• if ${\mathbf{order}}=\mathrm{Nag_ColMajor}$, ${\mathbf{pdx}}\ge {\mathbf{n}}$;
• if ${\mathbf{order}}=\mathrm{Nag_RowMajor}$, ${\mathbf{pdx}}\ge {\mathbf{m}}$.
6:     isx[m]const IntegerInput
On entry: indicates which $m$ independent variables are included in the model.
${\mathbf{isx}}\left[j-1\right]=1$
The $j$th variable in x will be included in the model.
${\mathbf{isx}}\left[j-1\right]=0$
Variable $j$ is excluded.
Constraint: ${\mathbf{isx}}\left[\mathit{j}-1\right]=0\text{​ or ​}1$, for $\mathit{j}=1,2,\dots ,{\mathbf{m}}$.
7:     ipIntegerInput
On entry: $m$, the number of independent variables in the model.
Constraints:
• $1\le {\mathbf{ip}}\le {\mathbf{m}}$;
• Exactly ip elements of isx must be equal to $1$.
8:     y[n]const doubleInput
On entry: the $n$ values of the dependent variable $y$.
9:     lhIntegerInput
On entry: the number of supplied ridge parameters.
Constraint: ${\mathbf{lh}}>0$.
10:   h[lh]const doubleInput
On entry: ${\mathbf{h}}\left[j-1\right]$ is the value of the $j$th ridge parameter $h$.
Constraint: ${\mathbf{h}}\left[\mathit{j}-1\right]\ge 0.0$, for $\mathit{j}=1,2,\dots ,{\mathbf{lh}}$.
11:   nep[lh]doubleOutput
On exit: ${\mathbf{nep}}\left[\mathit{j}-1\right]$ is the number of effective parameters, $\gamma$, in the $\mathit{j}$th model, for $\mathit{j}=1,2,\dots ,{\mathbf{lh}}$.
12:   wantbNag_ParaOptionInput
On entry: defines the options for parameter estimates.
${\mathbf{wantb}}=\mathrm{Nag_NoPara}$
Parameter estimates are not calculated and b is not referenced.
${\mathbf{wantb}}=\mathrm{Nag_OrigPara}$
Parameter estimates $b$ are calculated for the original data.
${\mathbf{wantb}}=\mathrm{Nag_StandPara}$
Parameter estimates $\stackrel{~}{b}$ are calculated for the standardized data.
Constraint: ${\mathbf{wantb}}=\mathrm{Nag_NoPara}$, $\mathrm{Nag_OrigPara}$ or $\mathrm{Nag_StandPara}$.
13:   b[$\mathit{dim}$]doubleOutput
Note: the dimension, dim, of the array b must be at least
• ${\mathbf{pdb}}×{\mathbf{lh}}$ when ${\mathbf{wantb}}\ne \mathrm{Nag_NoPara}$ and ${\mathbf{order}}=\mathrm{Nag_ColMajor}$;
• $\mathrm{max}\phantom{\rule{0.125em}{0ex}}\left(1,\left({\mathbf{ip}}+1\right)×{\mathbf{pdb}}\right)$ when ${\mathbf{wantb}}\ne \mathrm{Nag_NoPara}$ and ${\mathbf{order}}=\mathrm{Nag_RowMajor}$;
• $1$ otherwise.
Where ${\mathbf{B}}\left(i,j\right)$ appears in this document, it refers to the array element
• ${\mathbf{b}}\left[\left(j-1\right)×{\mathbf{pdb}}+i-1\right]$ when ${\mathbf{order}}=\mathrm{Nag_ColMajor}$;
• ${\mathbf{b}}\left[\left(i-1\right)×{\mathbf{pdb}}+j-1\right]$ when ${\mathbf{order}}=\mathrm{Nag_RowMajor}$.
On exit: if ${\mathbf{wantb}}\ne \mathrm{Nag_NoPara}$, b contains the intercept and parameter estimates for the fitted ridge regression model in the order indicated by isx. ${\mathbf{B}}\left(1,\mathit{j}\right)$, for $\mathit{j}=1,2,\dots ,{\mathbf{lh}}$, contains the estimate for the intercept; ${\mathbf{B}}\left(\mathit{i}+1,j\right)$ contains the parameter estimate for the $\mathit{i}$th independent variable in the model fitted with ridge parameter ${\mathbf{h}}\left[j-1\right]$, for $\mathit{i}=1,2,\dots ,{\mathbf{ip}}$.
14:   pdbIntegerInput
On entry: the stride separating row or column elements (depending on the value of order) in the array b.
Constraints:
• if ${\mathbf{order}}=\mathrm{Nag_ColMajor}$,
• if ${\mathbf{wantb}}\ne \mathrm{Nag_NoPara}$, ${\mathbf{pdb}}\ge {\mathbf{ip}}+1$;
• otherwise ${\mathbf{pdb}}\ge 1$;
• if ${\mathbf{order}}=\mathrm{Nag_RowMajor}$,
• if ${\mathbf{wantb}}\ne \mathrm{Nag_NoPara}$, ${\mathbf{pdb}}\ge {\mathbf{lh}}$;
• otherwise ${\mathbf{pdb}}\ge 1$.
15:   wantvfNag_VIFOptionInput
On entry: defines the options for variance inflation factors.
${\mathbf{wantvf}}=\mathrm{Nag_NoVIF}$
Variance inflation factors are not calculated and the array vf is not referenced.
${\mathbf{wantvf}}=\mathrm{Nag_WantVIF}$
Variance inflation factors are calculated.
Constraints:
• ${\mathbf{wantvf}}=\mathrm{Nag_NoVIF}$ or $\mathrm{Nag_WantVIF}$;
• if ${\mathbf{wantb}}=\mathrm{Nag_NoPara}$, ${\mathbf{wantvf}}=\mathrm{Nag_WantVIF}$.
16:   vf[$\mathit{dim}$]doubleOutput
Note: the dimension, dim, of the array vf must be at least
• ${\mathbf{pdvf}}×{\mathbf{lh}}$ when ${\mathbf{wantvf}}\ne \mathrm{Nag_NoVIF}$ and ${\mathbf{order}}=\mathrm{Nag_ColMajor}$;
• $\mathrm{max}\phantom{\rule{0.125em}{0ex}}\left(1,{\mathbf{ip}}×{\mathbf{pdvf}}\right)$ when ${\mathbf{wantvf}}\ne \mathrm{Nag_NoVIF}$ and ${\mathbf{order}}=\mathrm{Nag_RowMajor}$;
• $1$ otherwise.
Where ${\mathbf{VF}}\left(i,j\right)$ appears in this document, it refers to the array element
• ${\mathbf{vf}}\left[\left(j-1\right)×{\mathbf{pdvf}}+i-1\right]$ when ${\mathbf{order}}=\mathrm{Nag_ColMajor}$;
• ${\mathbf{vf}}\left[\left(i-1\right)×{\mathbf{pdvf}}+j-1\right]$ when ${\mathbf{order}}=\mathrm{Nag_RowMajor}$.
On exit: if ${\mathbf{wantvf}}=\mathrm{Nag_WantVIF}$, the variance inflation factors. For the $\mathit{i}$th independent variable in a model fitted with ridge parameter ${\mathbf{h}}\left[j-1\right]$, ${\mathbf{VF}}\left(\mathit{i},j\right)$ is the value of ${v}_{\mathit{i}}$, for $\mathit{i}=1,2,\dots ,{\mathbf{ip}}$.
17:   pdvfIntegerInput
On entry: the stride separating row or column elements (depending on the value of order) in the array vf.
Constraints:
• if ${\mathbf{order}}=\mathrm{Nag_ColMajor}$,
• if ${\mathbf{wantvf}}\ne \mathrm{Nag_NoVIF}$, ${\mathbf{pdvf}}\ge {\mathbf{ip}}$;
• otherwise ${\mathbf{pdvf}}\ge 1$;
• if ${\mathbf{order}}=\mathrm{Nag_RowMajor}$,
• if ${\mathbf{wantvf}}\ne \mathrm{Nag_NoVIF}$, ${\mathbf{pdvf}}\ge {\mathbf{lh}}$;
• otherwise ${\mathbf{pdvf}}\ge 1$.
18:   lpecIntegerInput
On entry: the number of prediction error statistics to return; set ${\mathbf{lpec}}\le 0$ for no prediction error estimates.
19:   pec[lpec]const Nag_PredictErrorInput
On entry: if ${\mathbf{lpec}}>0$, ${\mathbf{pec}}\left[\mathit{j}-1\right]$ defines the $\mathit{j}$th prediction error, for $\mathit{j}=1,2,\dots ,{\mathbf{lpec}}$; otherwise pec is not referenced.
${\mathbf{pec}}\left[j-1\right]=\mathrm{Nag_BIC}$
Bayesian information criterion (BIC).
${\mathbf{pec}}\left[j-1\right]=\mathrm{Nag_FPE}$
Future prediction error (FPE).
${\mathbf{pec}}\left[j-1\right]=\mathrm{Nag_GCV}$
Generalized cross-validation (GCV).
${\mathbf{pec}}\left[j-1\right]=\mathrm{Nag_LOOCV}$
Leave-one-out cross-validation (LOOCV).
${\mathbf{pec}}\left[j-1\right]=\mathrm{Nag_EUV}$
Unbiased estimate of variance (UEV).
Constraint: if ${\mathbf{lpec}}>0$, ${\mathbf{pec}}\left[\mathit{j}-1\right]=\mathrm{Nag_BIC}$, $\mathrm{Nag_FPE}$, $\mathrm{Nag_GCV}$, $\mathrm{Nag_LOOCV}$ or $\mathrm{Nag_EUV}$, for $\mathit{j}=1,2,\dots ,{\mathbf{lpec}}$.
20:   pe[$\mathit{dim}$]doubleOutput
Note: the dimension, dim, of the array pe must be at least
• ${\mathbf{pdpe}}×{\mathbf{lh}}$ when ${\mathbf{lpec}}>0$ and ${\mathbf{order}}=\mathrm{Nag_ColMajor}$;
• $\mathrm{max}\phantom{\rule{0.125em}{0ex}}\left(1,{\mathbf{lpec}}×{\mathbf{pdpe}}\right)$ when ${\mathbf{lpec}}>0$ and ${\mathbf{order}}=\mathrm{Nag_RowMajor}$;
• $1$ otherwise.
Where ${\mathbf{PE}}\left(i,j\right)$ appears in this document, it refers to the array element
• ${\mathbf{pe}}\left[\left(j-1\right)×{\mathbf{pdpe}}+i-1\right]$ when ${\mathbf{order}}=\mathrm{Nag_ColMajor}$;
• ${\mathbf{pe}}\left[\left(i-1\right)×{\mathbf{pdpe}}+j-1\right]$ when ${\mathbf{order}}=\mathrm{Nag_RowMajor}$.
On exit: if ${\mathbf{lpec}}\le 0$, pe is not referenced; otherwise ${\mathbf{PE}}\left(\mathit{i},\mathit{j}\right)$ contains the prediction error of criterion ${\mathbf{pec}}\left[\mathit{i}-1\right]$ for the model fitted with ridge parameter ${\mathbf{h}}\left[\mathit{j}-1\right]$, for $\mathit{i}=1,2,\dots ,{\mathbf{lpec}}$ and $\mathit{j}=1,2,\dots ,{\mathbf{lh}}$.
21:   pdpeIntegerInput
On entry: the stride separating row or column elements (depending on the value of order) in the array pe.
Constraints:
• if ${\mathbf{order}}=\mathrm{Nag_ColMajor}$,
• if ${\mathbf{lpec}}>0$, ${\mathbf{pdpe}}\ge {\mathbf{lpec}}$;
• otherwise ${\mathbf{pdpe}}\ge 1$;
• if ${\mathbf{order}}=\mathrm{Nag_RowMajor}$,
• if ${\mathbf{lpec}}>0$, ${\mathbf{pdpe}}\ge {\mathbf{lh}}$;
• otherwise ${\mathbf{pdpe}}\ge 1$.
22:   failNagError *Input/Output
The NAG error argument (see Section 3.6 in the Essential Introduction).

6  Error Indicators and Warnings

NE_ALLOC_FAIL
Dynamic memory allocation failed.
On entry, argument $⟨\mathit{\text{value}}⟩$ had an illegal value.
NE_CONSTRAINT
On entry, ${\mathbf{wantb}}=\mathrm{Nag_NoPara}$ and ${\mathbf{wantvf}}=\mathrm{Nag_NoVIF}$.
NE_ENUM_INT_2
On entry, ${\mathbf{pdb}}=⟨\mathit{\text{value}}⟩$ and ${\mathbf{ip}}=⟨\mathit{\text{value}}⟩$.
Constraint: if ${\mathbf{wantb}}\ne \mathrm{Nag_NoPara}$, ${\mathbf{pdb}}\ge {\mathbf{ip}}+1$.
On entry, ${\mathbf{pdvf}}=⟨\mathit{\text{value}}⟩$ and ${\mathbf{ip}}=⟨\mathit{\text{value}}⟩$.
Constraint: if ${\mathbf{wantvf}}\ne \mathrm{Nag_NoVIF}$, ${\mathbf{pdvf}}\ge {\mathbf{ip}}$.
On entry, ${\mathbf{wantb}}=⟨\mathit{\text{value}}⟩$, ${\mathbf{pdb}}=⟨\mathit{\text{value}}⟩$, ${\mathbf{lh}}=⟨\mathit{\text{value}}⟩$.
Constraint: if ${\mathbf{wantb}}\ne \mathrm{Nag_NoPara}$, ${\mathbf{pdb}}\ge {\mathbf{lh}}$;
otherwise ${\mathbf{pdb}}\ge 1$.
On entry, ${\mathbf{wantvf}}=⟨\mathit{\text{value}}⟩$, ${\mathbf{pdvf}}=⟨\mathit{\text{value}}⟩$, ${\mathbf{lh}}=⟨\mathit{\text{value}}⟩$.
Constraint: if ${\mathbf{wantvf}}\ne \mathrm{Nag_NoVIF}$, ${\mathbf{pdvf}}\ge {\mathbf{lh}}$;
otherwise ${\mathbf{pdvf}}\ge 1$.
NE_INT
On entry, ${\mathbf{lh}}=⟨\mathit{\text{value}}⟩$.
Constraint: ${\mathbf{lh}}>0$.
On entry, ${\mathbf{n}}=⟨\mathit{\text{value}}⟩$.
Constraint: ${\mathbf{n}}\ge 1$.
NE_INT_2
On entry, ${\mathbf{m}}=⟨\mathit{\text{value}}⟩$ and ${\mathbf{n}}=⟨\mathit{\text{value}}⟩$.
Constraint: ${\mathbf{m}}\le {\mathbf{n}}$.
On entry, ${\mathbf{pdpe}}=⟨\mathit{\text{value}}⟩$ and ${\mathbf{lpec}}=⟨\mathit{\text{value}}⟩$.
Constraint: ${\mathbf{pdpe}}\ge {\mathbf{lpec}}$.
On entry, ${\mathbf{pdx}}=⟨\mathit{\text{value}}⟩$ and ${\mathbf{m}}=⟨\mathit{\text{value}}⟩$.
Constraint: ${\mathbf{pdx}}\ge {\mathbf{m}}$.
On entry, ${\mathbf{pdx}}=⟨\mathit{\text{value}}⟩$ and ${\mathbf{n}}=⟨\mathit{\text{value}}⟩$.
Constraint: ${\mathbf{pdx}}\ge {\mathbf{n}}$.
NE_INT_3
On entry, ${\mathbf{pdpe}}=⟨\mathit{\text{value}}⟩$, ${\mathbf{lpec}}=⟨\mathit{\text{value}}⟩$ and ${\mathbf{lh}}=⟨\mathit{\text{value}}⟩$.
Constraint: if ${\mathbf{lpec}}>0$, ${\mathbf{pdpe}}\ge {\mathbf{lh}}$;
otherwise ${\mathbf{pdpe}}\ge 1$.
NE_INT_ARG_CONS
ip does not equal the sum of elements in isx.
NE_INT_ARRAY_VAL_1_OR_2
On entry, ${\mathbf{isx}}\left[i-1\right]\ne 0$ or $1$ for at least one $i$.
NE_INTERNAL_ERROR
An internal error has occurred in this function. Check the function call and any array sizes. If the call is correct then please contact NAG for assistance.
NE_REAL_ARRAY_CONS
On entry, ${\mathbf{h}}\left[i-1\right]<0$ for at least one $i$.

7  Accuracy

The accuracy of nag_regsn_ridge (g02kbc) is closely related to that of the singular value decomposition.

8  Parallelism and Performance

nag_regsn_ridge (g02kbc) is threaded by NAG for parallel execution in multithreaded implementations of the NAG Library.
nag_regsn_ridge (g02kbc) makes calls to BLAS and/or LAPACK routines, which may be threaded within the vendor library used by this implementation. Consult the documentation for the vendor library for further information.

nag_regsn_ridge (g02kbc) allocates internally $\mathrm{max}\phantom{\rule{0.125em}{0ex}}\left(5×\left({\mathbf{n}}-1\right),2×{\mathbf{ip}}×{\mathbf{ip}}\right)+\left({\mathbf{n}}+3\right)×{\mathbf{ip}}+{\mathbf{n}}$ elements of double precision storage.

10  Example

This example reads in data from an experiment to model body fat, and a selection of ridge regression models are calculated.

10.1  Program Text

Program Text (g02kbce.c)

10.2  Program Data

Program Data (g02kbce.d)

10.3  Program Results

Program Results (g02kbce.r)