Apache SystemDS 2.1.0 Release Notes

Release notes for SystemDS 2.1.0

SystemDS 2.1 is a minor release. This release contains major improvements to existing features, a large number of fixes, and some experimental features to better support the end-to-end data science lifecycle. In addition to that, this is the last release with Java 8 and Spark 2.x.

The major changes (compared to SystemDS 2.0) include

  • Federated Backend: Extended operations and built-ins to support various workflows and algorithms. Extended parameter server for federated training of Neural Networks.
  • A wealth of new and improved built-in functions including support for string operations, time-series analysis, data cleaning, grid search hyperparameter tuning, multi-threaded feature transformations.
  • Better support for Eval and list data structure.
  • Various performance improvements in SystemDS’ local, distributed, and native runtimes.
  • Python API: Source functionality to allow arbitrary DML script, auto-generated Python API for all built-in functions, list support allowing Pytonic access and manipulation of lazily evaluated DML elements, multi return continuation, allowing continued execution past calls to builtin functions.
  • GPU Backend: Various performance improvements and fixes, cuda code refactor, cuda code generation [experimental].
  • Lineage Tracing and Reuse: Initial support for federated and GPU backends, new hybrid lineage deduplication and reuse of compressed lineage DAGs, lineage cache benefits estimator, improved cache eviction logic.
  • Usability: Better Log4j integration, debug flag to reduce error logging, support for remote debugging.
  • Miscellaneous Improvements: Improved documentation, better testing, streamlined release process.