naginterfaces.library.stat.quantiles_stream_arbitrary¶

naginterfaces.library.stat.quantiles_stream_arbitrary(ind, rv, nb, eps, q, nq, rcomm, icomm)[source]¶

quantiles_stream_arbitrary finds approximate quantiles from a large arbitrary-sized data stream using an out-of-core algorithm.

For full information please refer to the NAG Library document for g01ap

https://www.nag.com/numeric/nl/nagdoc_29.3/flhtml/g01/g01apf.html

Parameters

indint

On initial entry: must be set to $0$ .Indicates the action required in the current call to quantiles_stream_arbitrary.

$i n d = 0$

Initialize the communication arrays and attempt to process the first $n b$ values from the data stream. $e p s$ , $r v$ and $n b$ must be set and $licomm$ must be at least $10$ .

$i n d = 1$

Attempt to process the next block of $n b$ values from the data stream. The calling program must update $r v$ and (if required) $n b$ , and re-enter quantiles_stream_arbitrary with all other parameters unchanged.

$i n d = 2$

Continue calculation following the reallocation of either or both of the communication arrays $r c o m m$ and $i c o m m$ .

$i n d = 3$

Calculate the $n q$ $ϵ$ -approximate quantiles specified in $q$ . The calling program must set $q$ and $n q$ and re-enter quantiles_stream_arbitrary with all other parameters unchanged. This option can be chosen only when $n p \geq ⌈ e x p (1.0) / e p s ⌉$ .

rvfloat, array-like, shape $(:)$

Note: the required length for this argument is determined as follows: if $i n d in (0, 1, 2)$ : $n b$ ; otherwise: $0$ .

If $i n d = 0$ , $1$ or $2$ , the vector containing the current block of data, otherwise $r v$ is not referenced.

nbint

If $i n d = 0$ , $1$ or $2$ , the size of the current block of data. The size of blocks of data in array $r v$ can vary;, therefore, $n b$ can change between calls to quantiles_stream_arbitrary.

epsfloat

Approximation factor $ϵ$ .

qfloat, array-like, shape $(:)$

Note: the required length for this argument is determined as follows: if $i n d = 3$ : $n q$ ; otherwise: $0$ .

If $i n d = 3$ , the quantiles to be calculated, otherwise $q$ is not referenced. Note that $q [i] = 0.0$ , corresponds to the minimum value and $q [i] = 1.0$ to the maximum value.

nqint

If $i n d = 3$ , the number of quantiles requested, otherwise $n q$ is not referenced.

rcommfloat, ndarray, shape $(lrcomm)$ , modified in place

On entry: if $i n d = 1$ or $2$ then the first $l$ elements of $r c o m m$ as supplied to quantiles_stream_arbitrary must be identical to the first $l$ elements of $r c o m m$ returned from the last call to quantiles_stream_arbitrary, where $l$ is the value of $lrcomm$ used in the last call. In other words, the contents of $r c o m m$ must not be altered between calls to this function. If $r c o m m$ needs to be reallocated then its contents must be preserved. If $i n d = 0$ then $r c o m m$ need not be set.

On exit: $r c o m m$ holds information required by subsequent calls to quantiles_stream_arbitrary.

icommint, ndarray, shape $(licomm)$ , modified in place

On entry: if $i n d = 1$ or $2$ then the first $l$ elements of $i c o m m$ as supplied to quantiles_stream_arbitrary must be identical to the first $l$ elements of $i c o m m$ returned from the last call to quantiles_stream_arbitrary, where $l$ is the value of $licomm$ used in the last call. In other words, the contents of $i c o m m$ must not be altered between calls to this function. If $i c o m m$ needs to be reallocated then its contents must be preserved. If $i n d = 0$ then $i c o m m$ need not be set.

On exit: $i c o m m [0]$ holds the minimum required length for $r c o m m$ and $i c o m m [1]$ holds the minimum required length for $i c o m m$ . The remaining elements of $i c o m m$ are used for communication between subsequent calls to quantiles_stream_arbitrary.

Returns

indint

Indicates output from the call.

$i n d = 1$

quantiles_stream_arbitrary has processed $n p$ data points and expects to be called again with additional data.

$i n d = 2$

Either one or more of the communication arrays $r c o m m$ and $i c o m m$ is too small. The new minimum lengths of $r c o m m$ and $i c o m m$ have been returned in $i c o m m [0]$ and $i c o m m [1]$ respectively. If the new minimum length is greater than the current length then the corresponding communication array needs to be reallocated, its contents preserved and quantiles_stream_arbitrary called again with all other parameters unchanged.

If there is more data to be processed, it is recommended that $lrcomm$ and $licomm$ are made significantly bigger than the minimum to limit the number of reallocations.

$i n d = 3$

quantiles_stream_arbitrary has returned the requested $ϵ$ -approximate quantiles in $q v$ . These quantiles are based on $n p$ data points.

npint

$m$ , the number of elements processed so far.

qvfloat, ndarray, shape $(:)$

If $i n d = 3$ , $q v [i]$ contains the $ϵ$ -approximate quantiles specified by the value provided in $q [i]$ .

Raises

NagValueError

(errno $1$ )

On entry, $i n d = ⟨ v a l u e ⟩$ .

Constraint: $i n d = 0$ , $1$ , $2$ or $3$ .

(errno $2$ )

On entry, $e p s = ⟨ v a l u e ⟩$ .

Constraint: $0.0 < e p s \leq 1.0$ .

(errno $3$ )

On entry, $i n d = 0$ , $1$ or $2$ and $n b = ⟨ v a l u e ⟩$ .

Constraint: if $i n d = 0$ , $1$ or $2$ then $n b > 0$ .

(errno $6$ )

The contents of $i c o m m$ have been altered between calls to this function.

(errno $7$ )

The contents of $r c o m m$ have been altered between calls to this function.

(errno $8$ )

Number of data elements streamed, $⟨ v a l u e ⟩$ is not sufficient for a quantile query when $e p s = ⟨ v a l u e ⟩$ .

Supply more data or reprocess the data with a higher $e p s$ value.

(errno $9$ )

On entry, $i n d = 3$ and $n q = ⟨ v a l u e ⟩$ .

Constraint: if $i n d = 3$ then $n q > 0$ .

(errno $10$ )

On entry, $i n d = 3$ and $q [⟨ v a l u e ⟩] = ⟨ v a l u e ⟩$ .

Constraint: if $i n d = 3$ then $0.0 \leq q [i] \leq 1.0$ for all $i$ .

Notes

A quantile is a value which divides a frequency distribution such that there is a given proportion of data values below the quantile. For example, the median of a dataset is the $0.5$ quantile because half the values are less than or equal to it.

quantiles_stream_arbitrary uses a slightly modified version of an algorithm described in a paper by Zhang and Wang (2007) to determine $ϵ$ -approximate quantiles of a large arbitrary-sized data stream of real values, where $ϵ$ is a user-defined approximation factor. Let $m$ denote the number of data elements processed so far then, given any quantile $q \in [0.0, 1.0]$ , an $ϵ$ -approximate quantile is defined as an element in the data stream whose rank falls within $[(q - ϵ) m, (q + ϵ) m]$ . In case of more than one $ϵ$ -approximate quantile being available, the one closest to $q m$ is used.

References: Zhang, Q and Wang, W, 2007, A fast algorithm for approximate quantiles in high speed data streams, Proceedings of the 19th International Conference on Scientific and Statistical Database Management, IEEE Computer Society, 29

NAG and Python

Return to Front

naginterfaces.library.stat.quantiles_stream_arbitrary¶

naginterfaces.library.stat.quantiles_​stream_​arbitrary¶

naginterfaces.library.stat.quantiles_stream_arbitrary¶