The naginterfaces Python Package

Copyright:The Numerical Algorithms Group Limited, 2017-2018.

Introduction

The naginterfaces package is a set of Python interfaces for the Mark 26.2 NAG Library Engine, which is the software implementation of NAG’s collection of several hundred mathematical and statistical routines serving a diverse range of application areas.

NAG’s collection of algorithms is outlined in the Submodule Summary section of the documentation for the naginterfaces.library subpackage.

The package is compatible with Python 2.7, 3.4, 3.5, 3.6 and 3.7. See the Applicability section below for more details.

A selection of example files are included in the package distribution. These serve to demonstrate some important usage points and can also be used as short tests of the package.

The required NAG library and its appropriate runtime dependencies are included and installed with the package. You do not need to perform any installations of other off-the-shelf NAG Libraries to use this product. The included NAG library is customized for the package and cannot be used in any other context.

Two variants of the package are available, differing in the underlying linear algebra routines used by the NAG library: one set uses vendor-optimized routines and the other is ‘self contained’, which uses NAG-supplied versions.

In the front matter of the package’s documentation the following sections are available:

Changelog
The Changelog for the package, listing interesting enhancements and fixes introduced in each naginterfaces release.
Known Issues
An ‘issues’ document, describing known performance issues in the current release of the product.
Supplementary Information
Details of additional applicability or usage information for the product.
Technical Support
Information on how to receive further support from NAG.
Terms and Conditions
By installing this software you agree to NAG’s Terms and Conditions as outlined here.

Dependencies

  • A valid licence for this product. (See the Obtaining a Licence section below for further details.)
  • Python.
  • NumPy.

Installation

Installation is from a pre-compiled Python wheel archive via pip (Python’s package-management system) run from the command line. The licence-managed package is served from a NAG-hosted repository. Hence by using pip to install the software as documented in the Installing Using pip section below, the appropriate package for the system you are running pip on is downloaded and installed in one combined operation.

If you have a special need to interact directly with a source distribution of the package, how to do this is covered in the subsequent Source Distribution section.

By special arrangement a non-licence-managed .zip for the software can be obtained from a secure NAG URL.

The primary Python distribution supported by NAG is the official CPython distribution from

https://www.python.org

Use is also supported through Intel Python 2018

https://software.intel.com/en-us/articles/using-intel-distribution-for-python-with-anaconda

via Anaconda

https://www.continuum.io/

and using a NumPy enabled with Intel MKL version 2018.

Please take particular note that we do not support use with Anaconda’s default MKL-enabled NumPy (i.e., an MKL-enabled NumPy not obtained through the specific Anaconda channel for Intel Python) because of the potential runtime incompatibilities this can introduce; see also the Software-Stack Compatibility Notes section below.

pip is included by default in Python 2.7 from 2.7.9 and in Python 3 from 3.4. If you are using Python 2.7 older than 2.7.9 we recommend upgrading to the latest in the 2.7 series, or see

https://packaging.python.org/tutorials/installing-packages/

You must have permission to modify the directory tree of the Python interpreter with which you wish to install the package. Installation is often easiest when using a virtual Python environment, but doing so is not obligatory. See the Appendix section below for more details.

If you wish to add the package to an existing Python installation then we strongly advise that this installation should not contain the earlier NAG Python release nag4py, because the runtime systems used by this and naginterfaces are not mutually compatible. There is a high chance of unexpected behaviour if both packages are imported into the same Python session.

Installing Using pip

  • Licence-managed Package

    The installation command (using pip from the command line) looks like

    python -m pip install --extra-index-url https://www.nag.com/downloads/py/naginterfaces_mkl naginterfaces
    

    for installing the version of the package that relies on Intel MKL for optimized linear algebra routines, or

    python -m pip install --extra-index-url https://www.nag.com/downloads/py/naginterfaces_nag naginterfaces
    

    for the version that uses self-contained NAG-supplied linear algebra routines.

    Both URLs supply the package as a pre-compiled Python wheel archive applicable to the system you are running pip on.

  • Non-licence-managed Package

    If you have obtained a non-licence-managed .zip for the software, please refer to the Installing the Non-licence-managed Package section of the Appendix, below.

The prerequisite NumPy will be automatically installed from the Python Package Index if not already present.

To upgrade an existing installation of naginterfaces, add the --upgrade switch to the pip line

python -m pip install --upgrade --extra-index-url https://www.nag.com/downloads/py/naginterfaces_mkl naginterfaces

for the licence-managed package, for example.

Once installed by pip the package can later be uninstalled using

python -m pip uninstall naginterfaces

If at any point you wish to switch between the naginterfaces_nag and naginterfaces_mkl variants, since by design both variants supply a package with the same name (naginterfaces) you must uninstall the existing naginterfaces and then install the alternate. This can be done in one command by instructing pip to forceably reinstall, by using the --upgrade --force-reinstall switches.

The NAG package repository does not support searching using pip search, but the URLs

https://www.nag.com/downloads/py/naginterfaces_mkl/naginterfaces

and

https://www.nag.com/downloads/py/naginterfaces_nag/naginterfaces

both give simple lists of all versions available of the licence-managed distribution files.

Source Distribution

Because of the platform-specific libraries included in the package there is no conveniently-sized system-agnostic Python source distribution of naginterfaces that is installable by pip using the same scheme as above. To access a (licence-managed) naginterfaces source distribution you must locate the .zip for your target platform from the list at the URL

https://www.nag.com/downloads/py/naginterfaces_mkl/naginterfaces

or at

https://www.nag.com/downloads/py/naginterfaces_nag/naginterfaces

and then you can use pip to install from the file’s full URL, or download the file and run pip on the result, for example

python -m pip install naginterfaces-{ver}-{platform tag}.zip

The {ver} marker denotes the (pip compliant) version number of the package—for example, 26.2.0.1—and {platform tag} denotes a (pip compliant) platform indicator—for example, win-amd64 for 64-bit Windows.

The package’s embedded setup.py file will raise RuntimeError if the target system for installation is not supported or does not match the binary files distributed in the .zip archive.

Before Use

Following a successful deployment of the naginterfaces materials, it is recommended that you review the full documentation for the package before use. To access from a command prompt execute

python -m pydoc naginterfaces

or see the Package Documentation section.

The HTML documentation is the canonical documentation source for the package. It may have minor changes applied to it in place when these do not warrant a full re-release of the package.

Note that the presence of other libraries via settings in your environment might introduce runtime incompatibilities with naginterfaces. Please see the Software-Stack Compatibility Notes section below.

Obtaining a Licence

Obtaining a NAG licence may require you to communicate your ‘Kusari hostid’ to NAG. For convenience, the module

python -m naginterfaces.kusari

will determine whether a valid licence could be found, and if not it will display the Kusari hostid for you on Linux and Mac. On Windows it will launch a GUI form.

For more information on how to obtain a valid licence for the underlying NAG library see the kusari Submodule section.

Quick Start

Once you have successfully installed the package and obtained a valid licence (if using licence-managed software) we strongly advise that you study the full documentation for the package. These steps are covered in the preceding sections Installation, Obtaining a Licence and Before Use, respectively.

This Quick Start section serves to provide a short worked example of how to get started using the package via solving an optimization problem.

Suppose that we wish to minimize the ‘generalized Rosenbrock’ function using bound constrained optimization by quadratic approximations.

Interfaces to the NAG Library are provided in the naginterfaces.library subpackage:

>>> import naginterfaces.library as ni

(We use the convention that the >>> marker denotes the Python prompt.)

One can see from the HTML documentation at naginterfaces.library or by examining the output from

>>> help(ni)
Help on package naginterfaces.library in naginterfaces:

NAME
    naginterfaces.library
...
    Submodule Summary
    -----------------
    Python interfaces are available for the following algorithmic submodules.
...
    ``opt`` - Minimizing or Maximizing a Function (e04)

      This module provides functions for solving various mathematical
      optimization problems by solvers based on local stopping criteria.
...

that the relevant algorithmic submodule for (local) optimization is opt. (The opt Chapter is also known by its ‘short’ name e04 in the NAG Library Manual.) The HTML documentation for this submodule is at naginterfaces.library.opt.

In the naginterfaces HTML documentation all submodules provide a Functionality Index. Studying the opt Functionality Index, or alternatively viewing the output at the Python prompt after invoking

>>> from naginterfaces.library import opt as ni_opt; help(ni_opt)
Help on module naginterfaces.library.opt in naginterfaces.library:

...
FUNCTIONS
...
    bounds_bobyqa_func(objfun, npt, x, bl, bu, rhobeg, rhoend, maxcal, monfun=None, data=None)
        Bound constrained minimum, model-based algorithm, using function
        values only.
...
        Parameters
        ----------
        objfun : callable f = objfun(x, data=None)
...
        x : float, array-like, shape (n)
            An estimate of the position of the minimum. If any component is
            out-of-bounds it is replaced internally by the bound it
            violates.
...

confirms that the relevant optimization solver to call is bounds_bobyqa_func. The HTML documentation for this solver is at naginterfaces.library.opt.bounds_bobyqa_func().

(Information can also be discovered by using the search facility in the naginterfaces HTML documentation.)

The optimization solver may be imported directly if desired:

>>> from naginterfaces.library.opt import bounds_bobyqa_func

Now define the optimization problem: first, the objective function for the generalized Rosenbrock problem. From the signature displayed above we can infer that parameter objfun in bounds_bobyqa_func may be specified as a lambda expression in our case, where we do not have any communication data (data) to pass to the function:

>>> rosen = lambda x: (
...     sum(100.0*(x[1:]-x[:-1]**2.0)**2.0 + (1.0-x[:-1])**2.0)
... )

Then define an initial guess for the optimization. In the naginterfaces.library subpackage input array data may be supplied in any ‘array-like’ container, as noted above in the float, array-like, shape (n) specification for argument x. For our one-dimensional x, this means that any sequence of data will be a suitable container, as will a numpy.ndarray. (In functions taking multi-dimensional data, nested sequences and again instances of numpy.ndarray are valid.) Furthermore, the shape (length) of the x we supply determines the (inferred) value of n for the problem.

Our chosen start point is and thus any of the following may be used to supply the ‘array-like’ vector x:

  • as a list
>>> x = [1.2, 1.0, 1.2, 1.0]
  • as a tuple
>>> x = (1.2, 1.0, 1.2, 1.0)
  • as an ndarray
>>> import numpy as np; x = np.array([1.2, 1.0, 1.2, 1.0])

Now define box bounds for the problem:

>>> n = len(x)
>>> bl, bu = ([0.0]*n, [2.0]*n)

Initialize the remaining parameters for the optimization, which control the quadratic-approximation process, the size of the trust region used during the optimization, and the limit on iterations:

>>> npt = 2*n + 1
>>> rhobeg, rhoend = (1e-1, 1e-6)
>>> maxcal = 500

Minimize the problem:

>>> x_min, f, nf = bounds_bobyqa_func(
...     rosen, npt, x, bl, bu, rhobeg, rhoend, maxcal,
... )

Display the results:

>>> print('Function value at lowest point found is {:.5f}.'.format(f))
Function value at lowest point found is 0.00000.
>>> print('The objective function was called {:d} times.'.format(nf))
The objective function was called ... times.
>>> print('The corresponding x is (' +
...     ', '.join(['{:.4f}'] * n).format( *x_min ) +
... ').')
The corresponding x is (1.0000, 1.0000, 1.0000, 1.0000).

Example Files

Each example file for the package can be run as a Python module, invocable via for instance

python -m naginterfaces.library.examples.info.impl_details_ex

Running

python -m pydoc naginterfaces.library.examples.info.impl_details_ex

will confirm the installation location of the associated file. All examples can be run as standard Python ‘main’ scripts from the command prompt, by executing for instance

python path/to/naginterfaces/library/examples/info/impl_details_ex.py

Likewise, the package’s source distribution contains all the example files in subdirectories of naginterfaces/library/examples.

Python’s inspect module can be used to access the source code of objects:

>>> import inspect
>>> from naginterfaces.library.examples.opt import bounds_bobyqa_func_ex
>>> print(''.join(inspect.getsourcelines(bounds_bobyqa_func_ex)[0]))
#!/usr/bin/env python
"``naginterfaces.library.opt.bounds_bobyqa_func`` Python Example."
...

If your Python session is running in an Integrated Development Environment (IDE) such as IDLE, there may be a facility to open the source file for a requested module, for example naginterfaces.library.examples.info.impl_details_ex. Please refer to the documentation for your chosen IDE.

All examples may be executed sequentially by running

python -m naginterfaces.library.examples

Every example is executed using the doctest module, which will print details of any unexpected output.

To display the full list of example source files on disk, but not run them, execute

python -m naginterfaces.library.examples --locate

Run

python -m naginterfaces.library.examples --help

to see any additional usage.

Applicability

This package has been tested on the following systems:

  • 64-bit Linux (Fedora 27)
  • 64-bit Mac (macOS 10.12/Sierra and 10.13/High Sierra)
  • 64-bit Windows (Windows 10)

(We expect that Linux OS compatibility will be achieved by any system having GNU C Library version 2.12-1 or compatible.)

The Python distributions tested were

  • CPython

    • Python 2.7.15 (64-bit; Linux, Windows)
    • Python 3.7.0 (64-bit; all systems)

    both with NumPy

    • PyPI numpy 1.15
  • Intel Python 2018

    • Python 2.7.14 (64-bit; all systems)
    • Python 3.6.3 (64-bit; all systems)

    both with NumPy

    • Intel numpy 1.15 enabled with Intel MKL 2018

In particular, 32-bit Python is not supported. Please take care on Windows if installing CPython from https://www.python.org because the Python download suggested by the front page may be 32-bit Python, even on 64-bit Windows. The 64-bit Windows Python download can be found by following the link to the project’s downloads area and choosing the required Windows x86-64 installer.

Software-Stack Compatibility Notes

MKL-enabled NumPy

The only MKL-enabled NumPy which is known to be compatible with naginterfaces is that from the Intel Python distribution. The default MKL-enabled NumPy from Anaconda Python is hence not compatible.

A simple diagnostic of MKL incompatibility is failure of the example

python -m naginterfaces.library.examples.lapacklin.dgesv_ex

Runtime Libraries

The intention is for the package to be able to load the underlying NAG library and its compatible runtimes, which are side-by-side in the distribution, without you needing to make any explicit settings in your environment. However, if the underlying NAG library fails to load (and the package raises NagException) it may be that incompatible runtime libraries have been preloaded via settings in your environment; i.e., LD_LIBRARY_PATH on Linux, DYLD_LIBRARY_PATH on Mac, or PATH on Windows.

(The package’s exception hierarchy, which includes NagException, is documented in the naginterfaces.base.utils submodule.)

If you believe that your environment settings are interfering with the loading of the underlying NAG library please consider running your Python session with the other (e.g. NAG and Intel) settings excised from the environment.

You may also find that it helps to use the MKL-free naginterfaces_nag version of the naginterfaces package.

Alternatively, prepend the installed directory for the runtimes into the environment before beginning the Python session.

The installation directory can be displayed by executing

python -c "from naginterfaces import ENGINE_LIBDIR; print(ENGINE_LIBDIR)"

To add the displayed directory into the environment: for Bourne shell users on Linux

LD_LIBRARY_PATH=path/to/naglib_dir:"${LD_LIBRARY_PATH}"
export LD_LIBRARY_PATH

for example, or for Bourne shell users on Mac

DYLD_LIBRARY_PATH=path/to/naglib_dir:"${DYLD_LIBRARY_PATH}"
export DYLD_LIBRARY_PATH

or from a DOS prompt on Windows

PATH=path\to\naglib_dir;%PATH%

Deadlock in Intel 2018 OpenMP

There is a known issue with fork() in Intel OpenMP applications:

https://software.intel.com/en-us/forums/intel-c-compiler/topic/758961

This issue affects potentially any Python code on Linux or Mac that contains calls to Intel MKL 2018 and the system fork(), directly or indirectly.

An example might be a Python script that calls an MKL-enabled NAG routine and then executes some other binary file using the subprocess module, which itself executes a system fork().

The symptom of the issue is that the code hangs. Inspecting using strace on Linux shows repeated calls to sched_yield().

A workaround suggested by Intel is to set environment variable KMP_INIT_AT_FORK to FALSE before running the Python process. Alternatively, NAG suggest using the MKL-free naginterfaces_nag version of the naginterfaces package if you believe you are affected by this, or indeed any, MKL-related issue.

Relaxed Stride Checking in NumPy

NumPy from version 1.12 onward treats ‘spiked’ arrays as being contiguous in both the Fortran and C senses, whereas in earlier releases they were only deemed C contiguous.

This affects the array-storage validation done by naginterfaces when used with an older NumPy and in the presence of ‘spiked’ arrays.

For example, x in the statement

x = numpy.array([1, 2, 3, 4]).reshape((1, 4))

when using NumPy older than 1.12 is considered by naginterfaces to only be compatible with C contiguous arrays, but when using NumPy 1.12 and newer it is considered compatible with arrays that use either storage scheme.

We therefore recommend using NumPy 1.12 or newer for better performance and usability in this regard.

TLS Compatibility with PyPI

Downloading packages (e.g. NumPy) from PyPI requires a Python that uses TLS protocol 1.2. Most Python distributions are suitable in this regard, but not so for CPython 2.7 on Mac.

On Mac we recommend using CPython 3 or Intel Python of any applicable version as given above.

Appendix

Creating a New Python Environment

Python virtual environments can be created using the virtualenv module for Python 2 or the venv module for Python 3. The Python 2 virtualenv module is not a core module: in what follows, if using Python 2 we assume that you have already installed virtualenv.

For Python 2

python -m virtualenv ni2

or for Python 3

python3 -m venv ni3

say.

To use an existing Anaconda installation to create a new Intel Python environment including an MKL-enabled NumPy, with conda on your path invoke one of the following forms, depending on the Python version required.

For Python 2

conda create --channel intel --prefix ni2 python=2 intelpython=2018 mkl=2018 numpy

or for Python 3

conda create --channel intel --prefix ni3 python=3 intelpython=2018 mkl=2018 numpy

You can then ‘activate’ the resulting Python environment.

Environments created using virtualenv or venv can be activated as follows: for Bourne shell users on Linux and macOS

. ni2/bin/activate

say, or on Windows

ni2\Scripts\activate

For Anaconda environments, under the Bash or Zsh shells (Linux and macOS)

source activate ni2

or on Windows

activate ni2

for example.

Environments can be deactivated in a similar way. For virtualenv and venv environments run

deactivate

For Anaconda environments, on Linux and macOS

source deactivate

or on Windows

deactivate

Obtaining the NAG Library Manual for Offline Access

A downloadable archive of the Library Manual to accompany this release can be found at

https://www.nag.com/numeric/fl/nagdoc_26.2/nagdoc_26.2.zip

Installing the Non-licence-managed Package

This section is relevant only when NAG has supplied you with a link to download a .zip for a non-licence-managed version of the package. Please read this section in conjunction with the main Installation section above.

The .zip file to which you are given access recreates the structure of the NAG package repository used by the licence-managed software, for each supported platform. Therefore you can emulate the same hosting of the package on your own systems if desired.

The name of the archive you access will look something like naginterfaces-{ver}-{plat}.zip, with {ver} being the package’s version number (for example, 26.2.0.1) and {plat} a platform indicator (for example, win for Windows).

Once the archive is unzipped, the wheel (.whl) installer for the MKL-enabled variant of the package can be found in

naginterfaces-{ver}-{plat}/naginterfaces_mkl/naginterfaces/

Likewise, the variant of the package that uses NAG-supplied linear algebra can be found in

naginterfaces-{ver}-{plat}/naginterfaces_nag/naginterfaces/

The wheel you wish to use can be installed directly from the local disk with pip. The name of the wheel file contains pip compliant markers indicating amongst other things the applicable Python versions for the package as well as its target platform. It will be of the form

naginterfaces-{ver}-{python tag}-none-{platform tag}.whl

Call this {ni_whl}. Each naginterfaces subdirectory of the unzipped archive will only contain one such wheel.

Therefore, the command for installing the MKL-enabled package will look like

python -m pip install naginterfaces-{ver}-{plat}/naginterfaces_mkl/naginterfaces/{ni_whl}

and for installing the package with NAG-supplied linear algebra, like:

python -m pip install naginterfaces-{ver}-{plat}/naginterfaces_nag/naginterfaces/{ni_whl}

The unzipped archive also contains .zip files for the package’s source distributions, in the same directories as the wheels. These source distributions have names like naginterfaces-{ver}-{platform tag}.zip.