SystemDS Install from source

Developing Apache SystemDS on Windows Platform

These instructions will help you build Apache SystemDS from source code, which is the basis for the engine and algorithms development. The following conventions will be used to refer to directories on your machine:

Directory structure of the installation

├── MAVEN_HOME = './maven'
|   ├── bin
|       ├── mvn.exe # and other executables
├── SPARK_HOME = './spark'
|   ├── bin
|       ├── spark-shell # and other invocation commands
├── HADOOP_HOME = './hadoop'
   ├── bin
|       ├── wintuils.exe # The Default content layout and html file.

Getting Apache SystemDS Source Code

SystemDS source code is available from by either cloning or downloading a zip file (based on a branch) into <SYSTEMDS_HOME>. The default is the main branch.

git clone systemds

The main branch contains the source code which will be used to create the next major version of Apache SystemDS.

Speed Tip: If the complete repository history isn’t needed then using a shallow clone (git clone --depth 1) will save significant time.

Building SystemDS source code

IntelliJ IDEA or Eclipse is preferred for best developer experience.

Opening the IntelliJ Source Code for Build

Using IntelliJ IDEA **File Open**, select the <SYSTEMDS_HOME> directory.

IntelliJ Build Configuration

JDK version 1.8 (u151 or newer) is required for building and developing for SystemDS developement.

  1. Using IntelliJ IDEA, configure a JDK named “1.8”, pointing to <JDK_18_HOME>.
    • If not already present, add <JDK_18_HOME>/lib/tools.jar to the Classpath tab for the 1.8 JDK.
  2. If the Maven Integration plugin is disabled, add the path variableMAVEN_REPOSITORY” pointing to <USER_HOME>/.m2/repository directory.
  3. Speed Tip: If you have enough RAM on your computer, configure the compiler settings to enable the “Compile independent modules in parallel” option. Also set the “User-local build process VM options” to -Xmx2G. These changes will greatly reduce the compile time.
  4. Now, selecting the IntelliJ IDEA **Build Build module ‘systemds’** option starts the maven build.
  5. Speed Tip:_ If the development machine have enough RAM, configure the compiler settings to enable the “Compile independent modules in parallel” option.

Building the Source Code

To build SystemDS from source, choose Build | Build Project from the main menu. OR To maven build, run the mvn clean package command in <SYSTEMDS_HOME> directory. See the pom.xml file for details.


To run the SystemDS built from source, choose **Run Run** from the main menu.
To run tests on the build, apply these setting to the **Run Edit Configurations… Defaults JUnit** configuration tab: