org.apache.sysds.runtime.controlprogram.parfor

Class RemoteParForSpark

java.lang.Object

org.apache.sysds.runtime.controlprogram.parfor.RemoteParForSpark

public class RemoteParForSpark
extends Object

This class serves two purposes: (1) isolating Spark imports to enable running in environments where no Spark libraries are available, and (2) to follow the same structure as the parfor remote_mr job submission. NOTE: currently, we still exchange inputs and outputs via hdfs (this covers the general case if data already resides in HDFS, in-memory data, and partitioned inputs; also, it allows for pre-aggregation by overwriting partial task results with pre-paggregated results from subsequent iterations) TODO reducebykey on variable names

Constructor Summary

Constructors

Constructor and Description
RemoteParForSpark()

Method Summary

Modifier and Type	Method and Description
static RemoteParForJobReturn	runJob(long pfid, String prog, HashMap<String,byte[]> clsMap, List<Task> tasks, ExecutionContext ec, ArrayList<ParForStatementBlock.ResultVar> resultVars, boolean cpCaching, int numMappers, boolean topLevelPF)

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RemoteParForSpark

public RemoteParForSpark()

Method Detail

runJob

public static RemoteParForJobReturn runJob(long pfid,
                                           String prog,
                                           HashMap<String,byte[]> clsMap,
                                           List<Task> tasks,
                                           ExecutionContext ec,
                                           ArrayList<ParForStatementBlock.ResultVar> resultVars,
                                           boolean cpCaching,
                                           int numMappers,
                                           boolean topLevelPF)