Native Backend guide for SystemDS

This guide covers the Native BLAS optimizations and software setup for using SystemDS native mode.

Native BLAS mode

SystemDS implements all the matrix operations in Java. This simplifies deployment especially in a distributed environment.

In Some cases (such as Deep Neural Networks), to take advantage of native BLAS instead of SystemDS internal Java library for performing single node operations such as matrix multiplication, convolution etc.

By default, SystemDS will first attempt to use Intel MKL (if installed), and then OpenBLAS (if installed). If none of the libraries are available, SystemDS falls back to its internal java library.

Note: Current SystemDS version supported on Linux, Windows x86-64 platform.

Requirements

Hardware

To know Intel MKL system requirements, see Intel® oneAPI Math Kernel Library System Requirements

Software

Either of the following software is required to be installed in your system:

INTEL MKL

Download Intel MKL. Supported MKL version is 2017 to 2019.5.

OpenBLAS

Install OpenBLAS. Installation instructions on GitHub.

Note: In Ubuntu 20.04, remove libopenblas0-pthread package and install libopenblas0-openmp instead. So that OpenBLAS will be installed with OpenMP support.

Native BLAS setup

Add library to the System path

Intel MKL

Scripts to set environmental variables

For example, in Linux to set Intel MKL libraries in the Path:

# path adjustments to that script according to your installation
source /opt/intel/bin/compilervars.sh intel64

OpenBLAS

Via commandline:

Java:

-Djava.library.path=/path/to/blas-n-other-dependencies

Note: This property can also be set with sysds.native.blas.directory.

Enable Native BLAS in SystemDS

Set sysds.native.blas property to mkl, openblas as shown.

<!-- enables native blas for matrix multiplication and convolution, experimental feature (options: auto, mkl, openblas, none) -->
    <sysds.native.blas>mkl</sysds.native.blas>

Troubleshooting

If there are issues loading libs because the gcc ABI changed from gcc 4.x to 5 and above. In this case,