Class Checkpoint


  • public class Checkpoint
    extends Lop
    Lop for checkpoint operations. For example, on Spark, the semantic of a checkpoint is to persist an intermediate result into a specific storage level (e.g., mem_only). We use the name checkpoint in terms of cache/persist in Spark (not in terms of checkpoint in Spark streaming) in order to differentiate from CP caching. NOTE: since this class uses spark apis, it should only be instantiated if we are running in execution mode spark (whenever all spark libraries are available)
    • Field Detail

      • DEFAULT_STORAGE_LEVEL

        public static final org.apache.spark.storage.StorageLevel DEFAULT_STORAGE_LEVEL
      • SER_STORAGE_LEVEL

        public static final org.apache.spark.storage.StorageLevel SER_STORAGE_LEVEL
      • CHECKPOINT_SPARSE_CSR

        public static final boolean CHECKPOINT_SPARSE_CSR
        See Also:
        Constant Field Values
    • Constructor Detail

      • Checkpoint

        public Checkpoint​(Lop input,
                          Types.DataType dt,
                          Types.ValueType vt,
                          String level,
                          boolean isAsync)
        TODO change string parameter storage.level to StorageLevel as soon as we can assume that Spark libraries are always available.
        Parameters:
        input - low-level operator
        dt - data type
        vt - value type
        level - storage level
        isAsync - true if eager and asynchronous checkpoint
    • Method Detail

      • getStorageLevel

        public org.apache.spark.storage.StorageLevel getStorageLevel()
      • setStorageLevel

        public void setStorageLevel​(org.apache.spark.storage.StorageLevel level)
      • toString

        public String toString()
        Description copied from class: Lop
        Method to have Lops print their state. This is for debugging purposes.
        Specified by:
        toString in class Lop
      • getInstructions

        public String getInstructions​(String input1,
                                      String output)
        Description copied from class: Lop
        Method should be overridden if needed
        Overrides:
        getInstructions in class Lop
        Parameters:
        input1 - input 1
        output - output
        Returns:
        instructions as string
      • getStorageLevelString

        public static String getStorageLevelString​(org.apache.spark.storage.StorageLevel level)
        This is a utility method because Sparks StorageLevel.toString() is incompatible with its own fromString() method.
        Parameters:
        level - RDD storage level
        Returns:
        storage level as a string
      • getDefaultStorageLevelString

        public static String getDefaultStorageLevelString()
      • getSerializeStorageLevelString

        public static String getSerializeStorageLevelString()