NAG CL Interface
h05aac (best_​subset_​given_​size_​revcomm)

Settings help

CL Name Style:


1 Purpose

Given a set of m features and a scoring mechanism for any subset of those features, h05aac selects the best n subsets of size p using a reverse communication branch and bound algorithm.

2 Specification

#include <nag.h>
void  h05aac (Integer *irevcm, Integer mincr, Integer m, Integer ip, Integer nbest, Integer *drop, Integer *lz, Integer z[], Integer *la, Integer a[], double bscore[], Integer bz[], Integer mincnt, double gamma, const double acc[], Integer icomm[], Integer licomm, double rcomm[], Integer lrcomm, NagError *fail)
The function may be called by the names: h05aac, nag_mip_best_subset_given_size_revcomm or nag_best_subset_given_size_revcomm.

3 Description

Given Ω={xi:i,1im}, a set of m unique features and a scoring mechanism f(S) defined for all S Ω then h05aac is designed to find So1 Ω , |So1| = p , an optimal subset of size p. Here |So1| denotes the cardinality of So1, the number of elements in the set.
The definition of the optimal subset depends on the properties of the scoring mechanism, if
f(Si) f(Sj) , for all ​ Sj Ω ​ and ​ Si Sj (1)
then the optimal subset is defined as one of the solutions to
maximize SΩ f(S)   subject to   |S| = p  
else if
f(Si) f(Sj) , for all ​ Sj Ω ​ and ​ Si Sj (2)
then the optimal subset is defined as one of the solutions to
minimize S Ω f(S)   subject to   |S| = p .  
If neither of these properties hold then h05aac cannot be used.
As well as returning the optimal subset, So1, h05aac can return the best n solutions of size p. If Soi denotes the ith best subset, for i=1,2,,n-1, then the (i+1)th best subset is defined as the solution to either
maximize S Ω - {Soj:j,1ji} f(S)   subject to   |S| = p  
or
minimize S Ω - {Soj:j,1ji} f(S)   subject to   |S| = p  
depending on the properties of f.
The solutions are found using a branch and bound method, where each node of the tree is a subset of Ω. Assuming that (1) holds then a particular node, defined by subset Si, can be trimmed from the tree if f(Si) < f^(Son) where f^(Son) is the nth highest score we have observed so far for a subset of size p, i.e., our current best guess of the score for the nth best subset. In addition, because of (1) we can also drop all nodes defined by any subset Sj where SjSi, thus avoiding the need to enumerate the whole tree. Similar short cuts can be taken if (2) holds. A full description of this branch and bound algorithm can be found in Ridout (1988).
Rather than calculate the score at a given node of the tree h05aac utilizes the fast branch and bound algorithm of Somol et al. (2004), and attempts to estimate the score where possible. For each feature, xi, two values are stored, a count ci and μ^i, an estimate of the contribution of that feature. An initial value of zero is used for both ci and μ^i. At any stage of the algorithm where both f(S) and f(S-{xi}) have been calculated (as opposed to estimated), the estimated contribution of the feature xi is updated to
ciμ^i + [f(S)-f(S-{xj})] ci+1  
and ci is incremented by 1, therefore, at each stage μ^i is the mean contribution of xi observed so far and ci is the number of observations used to calculate that mean.
As long as cik, for the user-supplied constant k, then rather than calculating f(S-{xi}) this function estimates it using f^(S-{xi}) = f(S) - γ μ^i or f^(S) - γ μ^i if f(S) has been estimated, where γ is a user-supplied scaling factor. An estimated score is never used to trim a node or returned as the optimal score.
Setting k=0 in this function will cause the algorithm to always calculate the scores, returning to the branch and bound algorithm of Ridout (1988). In most cases it is preferable to use the fast branch and bound algorithm, by setting k>0, unless the score function is iterative in nature, i.e., f(S) must have been calculated before f(S-{xi}) can be calculated.

4 References

Narendra P M and Fukunaga K (1977) A branch and bound algorithm for feature subset selection IEEE Transactions on Computers 9 917–922
Ridout M S (1988) Algorithm AS 233: An improved branch and bound algorithm for feature subset selection Journal of the Royal Statistics Society, Series C (Applied Statistics) (Volume 37) 1 139–147
Somol P, Pudil P and Kittler J (2004) Fast branch and bound algorithms for optimal feature selection IEEE Transactions on Pattern Analysis and Machine Intelligence (Volume 26) 7 900–912

5 Arguments

Note: this function uses reverse communication. Its use involves an initial entry, intermediate exits and re-entries, and a final exit, as indicated by the argument irevcm. Between intermediate exits and re-entries, all arguments other than bscore must remain unchanged.
1: irevcm Integer * Input/Output
On initial entry: must be set to 0.
On intermediate exit: irevcm=1 and before re-entry the scores associated with la subsets must be calculated and returned in bscore.
The la subsets are constructed as follows:
drop=1
The jth subset is constructed by dropping the features specified in the first lz elements of z and the single feature given in a[j-1] from the full set of features, Ω. The subset will, therefore, contain m-lz-1 features.
drop=0
The jth subset is constructed by adding the features specified in the first lz elements of z and the single feature specified in a[j-1] to the empty set, . The subset will, therefore, contain lz+1 features.
In both cases the individual features are referenced by the integers 1 to m with 1 indicating the first feature, 2 the second, etc., for some arbitrary ordering of the features. The same ordering must be used in all calls to h05aac.
If la=0, the score for a single subset should be returned. This subset is constructed by adding or removing only those features specified in the first lz elements of z.
If lz=0, this subset will either be Ω or .
The score associated with the jth subset must be returned in bscore[j-1].
On intermediate re-entry: irevcm must remain unchanged.
On final exit: irevcm=0, and the algorithm has terminated.
Constraint: irevcm=0 or 1.
Note: any values you return to h05aac as part of the reverse communication procedure should not include floating-point NaN (Not a Number) or infinity values, since these are not handled by h05aac. If your code inadvertently does return any NaNs or infinities, h05aac is likely to produce unexpected results.
2: mincr Integer Input
On entry: flag indicating whether the scoring function f is increasing or decreasing.
mincr=1
f(Si) f(Sj) , i.e., the subsets with the largest score will be selected.
mincr=0
f(Si) f(Sj) , i.e., the subsets with the smallest score will be selected.
For all SjΩ and SiSj.
Constraint: mincr=0 or 1.
3: m Integer Input
On entry: m, the number of features in the full feature set.
Constraint: m2.
4: ip Integer Input
On entry: p, the number of features in the subset of interest.
Constraint: 1ipm.
5: nbest Integer Input
On entry: n, the maximum number of best subsets required. The actual number of subsets returned is given by la on final exit. If on final exit lanbest then fail.code= NE_TOO_MANY is returned.
Constraint: nbest1.
6: drop Integer * Input/Output
On initial entry: drop need not be set.
On intermediate exit: flag indicating whether the intermediate subsets should be constructed by dropping features from the full set (drop=1) or adding features to the empty set (drop=0). See irevcm for details.
On intermediate re-entry: drop must remain unchanged.
On final exit: drop is undefined.
7: lz Integer * Input/Output
On initial entry: lz need not be set.
On intermediate exit: the number of features stored in z.
On intermediate re-entry: lz must remain unchanged.
On final exit: lz is undefined.
8: z[m-ip] Integer Input/Output
On initial entry: z need not be set.
On intermediate exit: z[i-1], for i=1,2,,lz, contains the list of features which, along with those specified in a, define the subsets whose score is required. See irevcm for additional details.
On intermediate re-entry: z must remain unchanged.
On final exit: z is undefined.
9: la Integer * Input/Output
On initial entry: la need not be set.
On intermediate exit: if la>0, the number of subsets for which a score must be returned.
If la=0, the score for a single subset should be returned. See irevcm for additional details.
On intermediate re-entry: la must remain unchanged.
On final exit: the number of best subsets returned.
10: a[max(nbest,m)] Integer Input/Output
On initial entry: a need not be set.
On intermediate exit: a[j-1], for j=1,2,,la, contains the list of features which, along with those specified in z, define the subsets whose score is required. See irevcm for additional details.
On intermediate re-entry: a must remain unchanged.
On final exit: a is undefined.
11: bscore[max(nbest,m)] double Input/Output
On initial entry: bscore need not be set.
On intermediate exit: bscore is undefined.
On intermediate re-entry: bscore[j-1] must hold the score for the jth subset as described in irevcm.
On final exit: holds the score for the la best subsets returned in bz.
12: bz[(m-ip)×nbest] Integer Input/Output
Note: where BZ(i,j) appears in this document, it refers to the array element bz[(j-1)×(m-ip)+i-1].
On initial entry: bz need not be set.
On intermediate exit: bz is used for storage between calls to h05aac.
On intermediate re-entry: bz must remain unchanged.
On final exit: the jth best subset is constructed by dropping the features specified in BZ(i,j), for i=1,2,,m-ip and j=1,2,,la, from the set of all features, Ω. The score for the jth best subset is given in bscore[j-1].
13: mincnt Integer Input
On entry: k, the minimum number of times the effect of each feature, xi, must have been observed before f(S-{xi}) is estimated from f(S) as opposed to being calculated directly.
If k=0 then f(S-{xi}) is never estimated. If mincnt<0 then k is set to 1.
14: gamma double Input
On entry: γ, the scaling factor used when estimating scores. If gamma<0 then γ=1 is used.
15: acc[2] const double Input
On entry: a measure of the accuracy of the scoring function, f.
Letting ai = ε1 |f(Si)| + ε2 , then when confirming whether the scoring function is strictly increasing or decreasing (as described in mincr), or when assessing whether a node defined by subset Si can be trimmed, then any values in the range f(Si) ± ai are treated as being numerically equivalent.
If 0acc[0]1 then ε1=acc[0], otherwise ε1=0.
If acc[1]0 then ε2=acc[1], otherwise ε2=0.
In most situations setting both ε1 and ε2 to zero should be sufficient. Using a nonzero value, when one is not required, can significantly increase the number of subsets that need to be evaluated.
16: icomm[licomm] Integer Communication Array
On initial entry: icomm need not be set.
On intermediate exit: icomm is used for storage between calls to h05aac.
On intermediate re-entry: icomm must remain unchanged.
On final exit: icomm is not defined. The first two elements, icomm[0] and icomm[1] contain the minimum required value for licomm and lrcomm respectively.
17: licomm Integer Input
On entry: the length of the array icomm. If licomm is too small and licomm2 then fail.code= NE_ARRAY_SIZE is returned and the minimum value for licomm and lrcomm are given by icomm[0] and icomm[1] respectively.
Constraints:
  • if mincnt=0, licomm 2× max(nbest,m)+ m(m+2)+ (m+1)× max(m-ip,1)+ 27 ;
  • otherwise licomm 2× max(nbest,m)+ m(m+3)+ (2m+1)× max(m-ip,1)+ 25 .
18: rcomm[lrcomm] double Communication Array
On initial entry: rcomm need not be set.
On intermediate exit: rcomm is used for storage between calls to h05aac.
On intermediate re-entry: rcomm must remain unchanged.
On final exit: rcomm is not defined.
19: lrcomm Integer Input
On entry: the length of the array rcomm. If lrcomm is too small and licomm2 then fail.code= NE_ARRAY_SIZE is returned and the minimum value for licomm and lrcomm are given by icomm[0] and icomm[1] respectively.
Constraints:
  • if mincnt=0, lrcomm 9+ nbest+ m× max(m-ip,1) ;
  • otherwise lrcomm 8+ m+ nbest+ m× max(m-ip,1) .
20: fail NagError * Input/Output
The NAG error argument (see Section 7 in the Introduction to the NAG Library CL Interface).

6 Error Indicators and Warnings

NE_ALLOC_FAIL
Dynamic memory allocation failed.
See Section 3.1.2 in the Introduction to the NAG Library CL Interface for further information.
NE_ARRAY_SIZE
On entry, licomm=value, lrcomm=value.
Constraint: licommvalue, lrcommvalue.
The minimum required values for licomm and lrcomm are returned in icomm[0] and icomm[1] respectively.
NE_BAD_PARAM
On entry, argument value had an illegal value.
NE_ILLEGAL_COMM
icomm has been corrupted between calls.
rcomm has been corrupted between calls.
NE_INT
On entry, irevcm=value.
Constraint: irevcm=0 or 1.
On entry, m=value.
Constraint: m2.
On entry, mincr=value.
Constraint: mincr=0 or 1.
On entry, nbest=value.
Constraint: nbest1.
NE_INT_2
On entry, ip=value and m=value.
Constraint: 1ipm.
NE_INT_CHANGED
drop has changed between calls.
On intermediate entry, drop=value.
On initial entry, drop=value.
ip has changed between calls.
On intermediate entry, ip=value.
On initial entry, ip=value.
la has changed between calls.
On entry, la=value.
On previous exit, la=value.
lz has changed between calls.
On entry, lz=value.
On previous exit, lz=value.
m has changed between calls.
On intermediate entry, m=value.
On initial entry, m=value.
mincnt has changed between calls.
On intermediate entry, mincnt=value.
On initial entry, mincnt=value.
mincr has changed between calls.
On intermediate entry, mincr=value.
On initial entry, mincr=value.
nbest has changed between calls.
On intermediate entry, nbest=value.
On initial entry, nbest=value.
NE_INTERNAL_ERROR
An internal error has occurred in this function. Check the function call and any array sizes. If the call is correct then please contact NAG for assistance.
See Section 7.5 in the Introduction to the NAG Library CL Interface for further information.
NE_NO_LICENCE
Your licence key may have expired or may not have been installed correctly.
See Section 8 in the Introduction to the NAG Library CL Interface for further information.
NE_REAL
bscore[value]=value, which is inconsistent with the score for the parent node. Score for the parent node is value.
NE_REAL_CHANGED
acc[0] has changed between calls.
On intermediate entry, acc[0]=value.
On initial entry, acc[0]=value.
acc[1] has changed between calls.
On intermediate entry, acc[1]=value.
On initial entry, acc[1]=value.
gamma has changed between calls.
On intermediate entry, gamma=value.
On initial entry, gamma=value.
NE_TOO_MANY
On entry, nbest=value.
But only value best subsets could be calculated.
NE_TOO_SMALL
On entry, licomm=value, lrcomm=value.
Constraint: licommvalue, lrcommvalue.
icomm is too small to return the required array sizes.

7 Accuracy

The subsets returned by h05aac are guaranteed to be optimal up to the accuracy of your calculated scores.

8 Parallelism and Performance

h05aac is threaded by NAG for parallel execution in multithreaded implementations of the NAG Library.
Please consult the X06 Chapter Introduction for information on how to control and interrogate the OpenMP environment used within this function. Please also consult the Users' Note for your implementation for any additional implementation-specific information.

9 Further Comments

The maximum number of unique subsets of size p from a set of m features is N= m! (m-p)!p! . The efficiency of the branch and bound algorithm implemented in h05aac comes from evaluating subsets at internal nodes of the tree, that is subsets with more than p features, and where possible trimming branches of the tree based on the scores at these internal nodes as described in Narendra and Fukunaga (1977). Because of this it is possible, in some circumstances, for more than N subsets to be evaluated. This will tend to happen when most of the features have a similar effect on the subset score.
If multiple optimal subsets exist with the same score, and nbest is too small to return them all, then the choice of which of these optimal subsets is returned is arbitrary.

10 Example

This example finds the three linear regression models, with five variables, that have the smallest residual sums of squares when fitted to a supplied dataset. The data used in this example was simulated.

10.1 Program Text

Program Text (h05aace.c)

10.2 Program Data

Program Data (h05aace.d)

10.3 Program Results

Program Results (h05aace.r)