SystemML Documentation

SystemML is now an Apache Incubator project! Please see the Apache SystemML website for more information.

SystemML is a flexible, scalable machine learning system. SystemML’s distinguishing characteristics are:

1. Algorithm customizability via R-like and Python-like languages.
2. Multiple execution modes, including Spark MLContext, Spark Batch, Hadoop Batch, Standalone, and JMLC.
3. Automatic optimization based on data and cluster characteristics to ensure both efficiency and scalability.

The SystemML GitHub README describes building, testing, and running SystemML. Please read Contributing to SystemML to find out how to help make SystemML even better!

To download SystemML, visit the downloads page.

Running SystemML

• Beginner’s Guide For Python Users - Beginner’s Guide for Python users.
• Spark MLContext - Spark MLContext is a programmatic API for running SystemML from Spark via Scala, Python, or Java.
  • Spark Shell Example (Scala)
  • Jupyter Notebook Example (PySpark)
• Spark Batch - Algorithms are automatically optimized to run across Spark clusters.
  • See Invoking SystemML in Spark Batch Mode for detailed information.
• Hadoop Batch - Algorithms are automatically optimized when distributed across Hadoop clusters.
  • See Invoking SystemML in Hadoop Batch Mode for detailed information.
• Standalone - Standalone mode allows data scientists to rapidly prototype algorithms on a single machine in R-like and Python-like declarative languages.
  • The Standalone Guide provides examples of algorithm execution in Standalone Mode.
• JMLC - Java Machine Learning Connector.
  • See Java Machine Learning Connector (JMLC) for more information.

Language Guides

• Python API Reference - API Reference Guide for Python users.
• DML Language Reference - DML is a high-level R-like declarative language for machine learning.
• PyDML Language Reference (Coming Soon) - PyDML is a high-level Python-like declarative language for machine learning.
• Beginner’s Guide to DML and PyDML - An introduction to the basics of DML and PyDML.

ML Algorithms

• Algorithms Reference - The Algorithms Reference describes the machine learning algorithms included with SystemML in detail.

Tools

• Debugger Guide - SystemML supports DML script-level debugging through a command-line interface.
• IDE Guide - Useful IDE Guide for Developing SystemML.

Other

• Contributing to SystemML - Describes ways to contribute to SystemML.
• Engine Developer Guide - Guide for internal SystemML engine development.
• Troubleshooting Guide - Troubleshoot various issues related to SystemML.
• Release Process - Description of the SystemML release process.