Apache SystemDS 3.0.0 Release Notes

Release notes for SystemDS 3.0.0

SystemDS 3.0 is a major release. This is the first release with Java 11 and Spark 3. Release 3.0 contains new features and major improvements to existing features.

The major changes (compared to SystemDS 2.1/2.2) include:

  • Federated Backend: Multi-tenancy support with tenants isolation and reuse of intermediates across tenants for federated workers. Compression support for federated workloads. Extended support for the cost-based federated planner, and various robustness and performance improvements.
  • Full support for Top-K cleaning framework – A framework for automatically generating the top-K most effective data cleaning pipelines.
  • Initial support for a Unified Memory Manager.
  • CUDA code generation and operator fusion [Experimental]
  • Robustness and performance improvements for Compressed Linear Algebra.
  • Performance and usability improvements for multi-threaded feature transformations.
  • Various new built-ins and apply functions for inference workloads.
  • Miscellaneous Improvements: Python end-to-end tutorials, improved documentation, and better testing