org.apache.sysds.runtime.controlprogram.parfor

Class RemoteParForUtils

java.lang.Object

org.apache.sysds.runtime.controlprogram.parfor.RemoteParForUtils

public class RemoteParForUtils
extends Object

Common functionalities for parfor workers in MR jobs. Used by worker wrappers in mappers (base RemoteParFor) and reducers (fused data partitioning and parfor)

Constructor Summary

Constructors

Constructor and Description
RemoteParForUtils()

Method Summary

Modifier and Type	Method and Description
static void	cleanupWorkingDirectories() Cleanup all temporary files created by this SystemDS process.
static void	cleanupWorkingDirectoriesOnShutdown() Cleanup all temporary files created by this SystemDS process, on shutdown via exit or interrupt.
static void	exportLineageItems(long workerID, LocalVariableMap vars, ArrayList<ParForStatementBlock.ResultVar> resultVars, Lineage lineage) Export lineage for remote Spark parfor workers.
static ArrayList<String>	exportResultVariables(long workerID, LocalVariableMap vars, ArrayList<ParForStatementBlock.ResultVar> resultVars) For remote Spark parfor workers.
static void	exportResultVariables(long workerID, LocalVariableMap vars, ArrayList<ParForStatementBlock.ResultVar> resultVars, HashMap<String,String> rvarFnames, org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.Writable,org.apache.hadoop.io.Writable> out) For remote MR parfor workers.
static void	exportResultVariables(long workerID, LocalVariableMap vars, ArrayList<ParForStatementBlock.ResultVar> resultVars, org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.Writable,org.apache.hadoop.io.Writable> out)
static Lineage[]	getLineages(LocalVariableMap[] results)
static LocalVariableMap[]	getResults(List<scala.Tuple2<Long,String>> out, org.apache.commons.logging.Log LOG)
static void	incrementParForMRCounters(org.apache.hadoop.mapred.Reporter reporter, long deltaTasks, long deltaIterations)
static void	setupBufferPool(long workerID) Init and register-cleanup of buffer pool

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RemoteParForUtils

public RemoteParForUtils()

Method Detail

incrementParForMRCounters

public static void incrementParForMRCounters(org.apache.hadoop.mapred.Reporter reporter,
                                             long deltaTasks,
                                             long deltaIterations)

exportResultVariables

public static void exportResultVariables(long workerID,
                                         LocalVariableMap vars,
                                         ArrayList<ParForStatementBlock.ResultVar> resultVars,
                                         org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.Writable,org.apache.hadoop.io.Writable> out)
                                  throws IOException

Throws:

IOException

exportResultVariables

public static void exportResultVariables(long workerID,
                                         LocalVariableMap vars,
                                         ArrayList<ParForStatementBlock.ResultVar> resultVars,
                                         HashMap<String,String> rvarFnames,
                                         org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.Writable,org.apache.hadoop.io.Writable> out)
                                  throws IOException

For remote MR parfor workers.

Parameters:

workerID - worker id

vars - local variable map

resultVars - list of result variables

rvarFnames - ?

out - output collectors

Throws:

IOException - if IOException occurs

exportResultVariables

public static ArrayList<String> exportResultVariables(long workerID,
                                                      LocalVariableMap vars,
                                                      ArrayList<ParForStatementBlock.ResultVar> resultVars)
                                               throws IOException

For remote Spark parfor workers. This is a simplified version compared to MR.

Parameters:

workerID - worker id

vars - local variable map

resultVars - list of result variables

Returns:

list of result variables

Throws:

IOException - if IOException occurs

exportLineageItems

public static void exportLineageItems(long workerID,
                                      LocalVariableMap vars,
                                      ArrayList<ParForStatementBlock.ResultVar> resultVars,
                                      Lineage lineage)
                               throws IOException

Export lineage for remote Spark parfor workers.

Parameters:

workerID - worker id

vars - local variable map

resultVars - list of result variables

lineage - lineage object

Throws:

IOException - if IOException occurs

cleanupWorkingDirectories

public static void cleanupWorkingDirectories()

Cleanup all temporary files created by this SystemDS process.

cleanupWorkingDirectoriesOnShutdown

public static void cleanupWorkingDirectoriesOnShutdown()

Cleanup all temporary files created by this SystemDS process, on shutdown via exit or interrupt.

getLineages

public static Lineage[] getLineages(LocalVariableMap[] results)

getResults

public static LocalVariableMap[] getResults(List<scala.Tuple2<Long,String>> out,
                                            org.apache.commons.logging.Log LOG)

setupBufferPool

public static void setupBufferPool(long workerID)
                            throws IOException

Init and register-cleanup of buffer pool

Parameters:

workerID - worker id

Throws:

IOException - exception