Package org.apache.sysds.hops

Class DataOp

java.lang.Object

org.apache.sysds.hops.Hop

org.apache.sysds.hops.DataOp

All Implemented Interfaces:

ParseInfo

public class DataOp
extends Hop

A DataOp can be either a persistent read/write or transient read/write - writes will always have at least one input, but all types can have parameters (e.g., for csv literals of delimiter, header, etc).

Field Summary

Fields inherited from class org.apache.sysds.hops.Hop

_beginColumn, _beginLine, _endColumn, _endLine, _filename, _text, CPThreshold

Constructor Summary

Constructors

Constructor	Description
DataOp(String l, Types.DataType dt, Types.ValueType vt, Types.OpOpData dop, String fname, long dim1, long dim2, long nnz, int blen)	READ operation for Matrix w/ dim1, dim2.
DataOp(String l, Types.DataType dt, Types.ValueType vt, Types.OpOpData dop, String fname, long dim1, long dim2, long nnz, MatrixObject.UpdateType update, int blen)
DataOp(String l, Types.DataType dt, Types.ValueType vt, Types.OpOpData dop, HashMap<String,Hop> params)	READ operation for Matrix / SQL operation for tensor This constructor supports expressions in parameters
DataOp(String l, Types.DataType dt, Types.ValueType vt, Types.OpOpData dop, Hop in, HashMap<String,Hop> inputParameters)	WRITE operation for Matrix This constructor supports expression in parameters
DataOp(String l, Types.DataType dt, Types.ValueType vt, Hop in, Types.OpOpData dop, String fname)

Method Summary

Modifier and Type	Method	Description
boolean	allowsAllExecTypes()
void	checkArity()	Check for N (READ) or N+1 (WRITE) inputs.
Object	clone()
boolean	compare(Hop that)
Lop	constructLops()
void	disableRecompileRead()	Explicitly disables recompilation of transient reads, this additional information is required because requiresRecompile is set in a top-down manner, hence any value set from a consuming operating would be overwritten by opFindExecType.
Types.FileFormat	getFileFormat()
String	getFileName()
long	getInputBlocksize()
Types.OpOpData	getOp()
String	getOpString()
int	getParameterIndex(String name)
boolean	hasOnlyRDD()
boolean	isFederatedDataOp()	Checks if the hop is a DataOp with federated data.
boolean	isGPUEnabled()	In memory-based optimizer mode (see OptimizerUtils.isMemoryBasedOptLevel()), the exectype is determined by checking this method as well as memory budget of this Hop.
boolean	isPersistentReadWrite()
boolean	isRead()
boolean	isWrite()
void	refreshSizeInformation()	Update the output size information for this hop.
void	removeInput(String inputName)	Remove an input from the list of inputs and from the parameter index map.
void	setDataOpType(Types.OpOpData type)
void	setFileFormat(Types.FileFormat ft)
void	setFileName(String fn)
void	setInputBlocksize(long blen)
void	setOnlyRDD(boolean flag)
void	setOutputParams(long dim1, long dim2, long nnz, MatrixObject.UpdateType update, int blen)

Methods inherited from class org.apache.sysds.hops.Hop

activatePrefetch, addAllInputs, addInput, checkAndSetForcedPlatform, checkAndSetInvalidCPDimsAndSize, clearMemEstimate, colsKnown, compressedSize, computeBoundsInformation, computeBoundsInformation, computeBoundsInformation, computeMemEstimate, computeSizeInformation, computeSizeInformation, computeSizeInformation, constructAndSetLopsDataFlowProperties, createOffsetLop, deactivatePrefetch, dimsKnown, dimsKnown, dimsKnownAny, federatedCostInitialized, getBeginColumn, getBeginLine, getBlocksize, getCompressedSize, getDataCharacteristics, getDataType, getDim, getDim1, getDim2, getEndColumn, getEndLine, getExecType, getFederatedCost, getFederatedOutput, getFilename, getForcedExecType, getHopID, getInput, getInput, getInputMemEstimate, getInputMemEstimate, getInputOutputSize, getIntermediateMemEstimate, getLength, getLops, getMemEstimate, getName, getNnz, getOutputMemEstimate, getOutputMemEstimate, getParent, getPrivacy, getRepetitions, getSparsity, getSpBroadcastSize, getText, getUpdateType, getValueType, hasCompressedInput, hasFederatedOutput, hasLocalOutput, hasMatrixInputWithDifferentBlocksizes, hasValidCPDimsAndSize, isCompressedOutput, isFederated, isMatrix, isMemEstimated, isOutputEmptyBlocks, isRequiredDecompression, isScalar, isTransposeSafe, isVisited, prefetchActivated, printErrorLocation, refreshColsParameterInformation, refreshColsParameterInformation, refreshMemEstimates, refreshRowsParameterInformation, refreshRowsParameterInformation, requiresCheckpoint, requiresCompression, requiresLineageCaching, requiresReblock, requiresRecompile, resetExecType, resetFederatedCost, resetRecompilationFlag, resetRecompilationFlag, resetVisitStatus, resetVisitStatus, resetVisitStatus, resetVisitStatusForced, rowsKnown, setBeginColumn, setBeginLine, setBlocksize, setCompressedOutput, setCompressedSize, setDataType, setDim, setDim1, setDim2, setEndColumn, setEndLine, setExecType, setFederatedCost, setFederatedOutput, setFilename, setForcedExecType, setLops, setMemEstimate, setName, setNnz, setOutputEmptyBlocks, setParseInfo, setPrivacy, setRequiresCheckpoint, setRequiresCompression, setRequiresCompression, setRequiresDeCompression, setRequiresLineageCaching, setRequiresReblock, setRequiresRecompile, setText, setUpdateType, setValueType, setVisited, setVisited, someInputFederated, toString, updateLopFedOut, updateLopFedOut, updateRepetitionEstimates

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

DataOp

public DataOp(String l,
              Types.DataType dt,
              Types.ValueType vt,
              Types.OpOpData dop,
              String fname,
              long dim1,
              long dim2,
              long nnz,
              int blen)

READ operation for Matrix w/ dim1, dim2. This constructor does not support any expression in parameters

Parameters:

l - ?

dt - data type

vt - value type

dop - data operator type

fname - file name

dim1 - dimension 1

dim2 - dimension 2

nnz - number of non-zeros

blen - rows/cols per block

DataOp

public DataOp(String l,
              Types.DataType dt,
              Types.ValueType vt,
              Types.OpOpData dop,
              String fname,
              long dim1,
              long dim2,
              long nnz,
              MatrixObject.UpdateType update,
              int blen)

DataOp

public DataOp(String l,
              Types.DataType dt,
              Types.ValueType vt,
              Types.OpOpData dop,
              HashMap<String,Hop> params)

READ operation for Matrix / SQL operation for tensor This constructor supports expressions in parameters

Parameters:

l - ?

dt - data type

vt - value type

dop - data operator type

params - input parameters

DataOp

public DataOp(String l,
              Types.DataType dt,
              Types.ValueType vt,
              Hop in,
              Types.OpOpData dop,
              String fname)

DataOp

public DataOp(String l,
              Types.DataType dt,
              Types.ValueType vt,
              Types.OpOpData dop,
              Hop in,
              HashMap<String,Hop> inputParameters)

WRITE operation for Matrix This constructor supports expression in parameters

Parameters:

l - ?

dt - data type

vt - value type

dop - data operator type

in - high-level operator

inputParameters - input parameters

Method Detail

checkArity

public void checkArity()

Check for N (READ) or N+1 (WRITE) inputs.

Specified by:

checkArity in class Hop

getOp

public Types.OpOpData getOp()

setDataOpType

public void setDataOpType(Types.OpOpData type)

setOutputParams

public void setOutputParams(long dim1,
                            long dim2,
                            long nnz,
                            MatrixObject.UpdateType update,
                            int blen)

setFileName

public void setFileName(String fn)

getFileName

public String getFileName()

getParameterIndex

public int getParameterIndex(String name)

setOnlyRDD

public void setOnlyRDD(boolean flag)

hasOnlyRDD

public boolean hasOnlyRDD()

isGPUEnabled

public boolean isGPUEnabled()

Description copied from class: Hop

In memory-based optimizer mode (see OptimizerUtils.isMemoryBasedOptLevel()), the exectype is determined by checking this method as well as memory budget of this Hop. Please see findExecTypeByMemEstimate for more detail. This method is necessary because not all operator are supported efficiently on GPU (for example: operations on frames and scalar as well as operations such as table).

Specified by:

isGPUEnabled in class Hop

Returns:

true if the Hop is eligible for GPU Exectype.

constructLops

public Lop constructLops()

Specified by:

constructLops in class Hop

setFileFormat

public void setFileFormat(Types.FileFormat ft)

getFileFormat

public Types.FileFormat getFileFormat()

setInputBlocksize

public void setInputBlocksize(long blen)

getInputBlocksize

public long getInputBlocksize()

isRead

public boolean isRead()

isWrite

public boolean isWrite()

isPersistentReadWrite

public boolean isPersistentReadWrite()

isFederatedDataOp

public boolean isFederatedDataOp()

Description copied from class: Hop

Checks if the hop is a DataOp with federated data.

Overrides:

isFederatedDataOp in class Hop

Returns:

true if hop is a federated DataOp

getOpString

public String getOpString()

Specified by:

getOpString in class Hop

allowsAllExecTypes

public boolean allowsAllExecTypes()

Specified by:

allowsAllExecTypes in class Hop

refreshSizeInformation

public void refreshSizeInformation()

Description copied from class: Hop

Update the output size information for this hop.

Specified by:

refreshSizeInformation in class Hop

disableRecompileRead

public void disableRecompileRead()

Explicitly disables recompilation of transient reads, this additional information is required because requiresRecompile is set in a top-down manner, hence any value set from a consuming operating would be overwritten by opFindExecType.

clone

public Object clone()
             throws CloneNotSupportedException

Specified by:

clone in class Hop

Throws:

CloneNotSupportedException

compare

public boolean compare(Hop that)

Specified by:

compare in class Hop

removeInput

public void removeInput(String inputName)

Remove an input from the list of inputs and from the parameter index map. Parameter index map values higher than the index of the removed input will be decremented by one.

Parameters:

inputName - The name of the input to remove