Hop (SystemDS 2.2.0-SNAPSHOT API)

java.lang.Object
- org.apache.sysds.hops.Hop

All Implemented Interfaces:

ParseInfo

Direct Known Subclasses:

DataOp, FunctionOp, IndexingOp, LeftIndexingOp, LiteralOp, MultiThreadedHop, NaryOp
```
public abstract class Hop
extends Object
implements ParseInfo
```

Field Summary

Fields
Modifier and Type Field and Description

int _beginColumn

int _beginLine

int _endColumn

int _endLine

String _filename

String _text

static long CPThreshold

Fields
Modifier and Type	Field and Description
`int`	`_beginColumn`
`int`	`_beginLine`
`int`	`_endColumn`
`int`	`_endLine`
`String`	`_filename`
`String`	`_text`
`static long`	`CPThreshold`

Constructor Summary

Constructors
Constructor and Description

Hop(String l, Types.DataType dt, Types.ValueType vt)

Constructors
Constructor and Description
`Hop(String l, Types.DataType dt, Types.ValueType vt)`

Method Summary

All Methods Static Methods Instance Methods Abstract Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`addAllInputs(ArrayList<Hop> list)`
`void`	`addInput(Hop h)`
`abstract boolean`	`allowsAllExecTypes()`
`void`	`checkAndSetForcedPlatform()`
`void`	`checkAndSetInvalidCPDimsAndSize()`
`abstract void`	`checkArity()` Check whether this Hop has a correct number of inputs.
`void`	`clearMemEstimate()`
`abstract Object`	`clone()`
`boolean`	`colsKnown()`
`abstract boolean`	`compare(Hop that)`
`long`	`compressedSize()`
`double`	`computeBoundsInformation(Hop input)`
`static double`	`computeBoundsInformation(Hop input, LocalVariableMap vars)`
`static double`	`computeBoundsInformation(Hop input, LocalVariableMap vars, HashMap<Long,Double> memo)`
`void`	`computeMemEstimate(MemoTable memo)` Computes the estimate of memory required to store the input/output of this hop in memory.
`static long`	`computeSizeInformation(Hop input)`
`long`	`computeSizeInformation(Hop input, LocalVariableMap vars)`
`long`	`computeSizeInformation(Hop input, LocalVariableMap vars, HashMap<Long,Long> memo)`
`void`	`constructAndSetLopsDataFlowProperties()`
`abstract Lop`	`constructLops()`
`static Lop`	`createOffsetLop(Hop hop, boolean repCols)`
`boolean`	`dimsKnown()`
`boolean`	`dimsKnown(boolean includeNnz)`
`boolean`	`dimsKnownAny()`
`int`	`getBeginColumn()`
`int`	`getBeginLine()`
`int`	`getBlocksize()`
`long`	`getCompressedSize()`
`DataCharacteristics`	`getDataCharacteristics()`
`Types.DataType`	`getDataType()`
`long`	`getDim(int i)`
`long`	`getDim1()`
`long`	`getDim2()`
`int`	`getEndColumn()`
`int`	`getEndLine()`
`Types.ExecType`	`getExecType()`
`String`	`getFilename()`
`Types.ExecType`	`getForcedExecType()`
`long`	`getHopID()`
`ArrayList<Hop>`	`getInput()`
`Hop`	`getInput(int ix)`
`double`	`getInputMemEstimate()`
`double`	`getInputOutputSize(Collection<String> exclVars)`
`double`	`getIntermediateMemEstimate()`
`long`	`getLength()`
`Lop`	`getLops()`
`double`	`getMemEstimate()` NOTES: * Purpose: Whenever the output dimensions / sparsity of a hop are unknown, this hop should store its worst-case output statistics (if known) in that table.
`String`	`getName()`
`long`	`getNnz()`
`abstract String`	`getOpString()`
`double`	`getOutputMemEstimate()`
`ArrayList<Hop>`	`getParent()`
`PrivacyConstraint`	`getPrivacy()`
`double`	`getSparsity()`
`double`	`getSpBroadcastSize()`
`String`	`getText()`
`MatrixObject.UpdateType`	`getUpdateType()`
`Types.ValueType`	`getValueType()`
`boolean`	`hasCompressedInput()`
`boolean`	`hasFederatedOutput()`
`boolean`	`hasMatrixInputWithDifferentBlocksizes()`
`boolean`	`hasValidCPDimsAndSize()`
`boolean`	`isCompressedOutput()`
`boolean`	`isFederated()`
`boolean`	`isFederatedOutput()`
`abstract boolean`	`isGPUEnabled()` In memory-based optimizer mode (see OptimizerUtils.isMemoryBasedOptLevel()), the exectype is determined by checking this method as well as memory budget of this Hop.
`boolean`	`isMatrix()`
`boolean`	`isMemEstimated()`
`boolean`	`isOutputEmptyBlocks()`
`boolean`	`isRequiredDecompression()`
`boolean`	`isScalar()`
`boolean`	`isTransposeSafe()` Defines if this operation is transpose-safe, which means that the result of op(input) is equivalent to op(t(input)).
`boolean`	`isVisited()`
`String`	`printErrorLocation()`
`void`	`refreshColsParameterInformation(Hop input, LocalVariableMap vars)`
`void`	`refreshColsParameterInformation(Hop input, LocalVariableMap vars, HashMap<Long,Long> memo)`
`void`	`refreshMemEstimates(MemoTable memo)` Recursively computes memory estimates for all the Hops in the DAG rooted at the current hop pointed by `this`.
`void`	`refreshRowsParameterInformation(Hop input, LocalVariableMap vars)`
`void`	`refreshRowsParameterInformation(Hop input, LocalVariableMap vars, HashMap<Long,Long> memo)`
`abstract void`	`refreshSizeInformation()` Update the output size information for this hop.
`boolean`	`requiresCheckpoint()`
`boolean`	`requiresCompression()`
`boolean`	`requiresLineageCaching()`
`boolean`	`requiresReblock()`
`boolean`	`requiresRecompile()` Indicates if dynamic recompilation is required for this hop.
`void`	`resetExecType()`
`static void`	`resetRecompilationFlag(ArrayList<Hop> hops, Types.ExecType et, Recompiler.ResetType reset)`
`static void`	`resetRecompilationFlag(Hop hops, Types.ExecType et, Recompiler.ResetType reset)`
`Hop`	`resetVisitStatus()`
`static void`	`resetVisitStatus(ArrayList<Hop> hops)`
`static void`	`resetVisitStatus(ArrayList<Hop> hops, boolean force)`
`void`	`resetVisitStatusForced(HashSet<Long> memo)`
`boolean`	`rowsKnown()`
`void`	`setBeginColumn(int passed)`
`void`	`setBeginLine(int passed)`
`void`	`setBlocksize(int blen)`
`void`	`setCompressedOutput(boolean value)`
`void`	`setCompressedSize(long size)`
`void`	`setDataType(Types.DataType dt)`
`void`	`setDim(int i, long dim)`
`void`	`setDim1(long dim1)`
`void`	`setDim2(long dim2)`
`void`	`setEndColumn(int passed)`
`void`	`setEndLine(int passed)`
`void`	`setExecType(Types.ExecType execType)`
`void`	`setFederatedOutput(FEDInstruction.FederatedOutput federatedOutput)`
`void`	`setFilename(String passed)`
`void`	`setForcedExecType(Types.ExecType etype)`
`void`	`setLops(Lop lops)`
`void`	`setMemEstimate(double mem)` Sets memory estimate in bytes
`void`	`setName(String _name)`
`void`	`setNnz(long nnz)`
`void`	`setOutputEmptyBlocks(boolean flag)`
`void`	`setParseInfo(ParseInfo parseInfo)` Set parse information.
`void`	`setPrivacy(PrivacyConstraint privacy)`
`void`	`setRequiresCheckpoint(boolean flag)`
`void`	`setRequiresCompression()`
`void`	`setRequiresCompression(AWTreeNode node)`
`void`	`setRequiresDeCompression()`
`void`	`setRequiresLineageCaching(boolean flag)`
`void`	`setRequiresReblock(boolean flag)`
`void`	`setRequiresRecompile()` Marks the hop for dynamic recompilation.
`void`	`setText(String text)`
`void`	`setUpdateType(MatrixObject.UpdateType update)`
`void`	`setValueType(Types.ValueType vt)`
`void`	`setVisited()`
`void`	`setVisited(boolean flag)`
`boolean`	`someInputFederated()`
`String`	`toString()`

Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.sysds.parser.ParseInfo
ctxAndFilenameToParseInfo

Field Detail

CPThreshold
```
public static final long CPThreshold
```
See Also:

Constant Field Values

_beginLine
```
public int _beginLine
```

_beginColumn
```
public int _beginColumn
```

_endLine
```
public int _endLine
```

_endColumn
```
public int _endColumn
```

_filename
```
public String _filename
```

_text
```
public String _text
```

Constructor Detail

Hop

public Hop(String l,
           Types.DataType dt,
           Types.ValueType vt)

Method Detail

getHopID
```
public long getHopID()
```

checkArity
```
public abstract void checkArity()
```
Check whether this Hop has a correct number of inputs. (Some Hops can have a variable number of inputs, such as DataOp, DataGenOp, ParameterizedBuiltinOp, ReorgOp, TernaryOp, QuaternaryOp, MultipleOp, DnnOp, and SpoofFusedOp.) Parameterized Hops (such as DataOp) can check that the number of parameters matches the number of inputs.

getExecType
```
public Types.ExecType getExecType()
```

setExecType

public void setExecType(Types.ExecType execType)

setFederatedOutput

public void setFederatedOutput(FEDInstruction.FederatedOutput federatedOutput)

resetExecType
```
public void resetExecType()
```

getForcedExecType

public Types.ExecType getForcedExecType()

setForcedExecType

public void setForcedExecType(Types.ExecType etype)

allowsAllExecTypes

public abstract boolean allowsAllExecTypes()

isTransposeSafe
```
public boolean isTransposeSafe()
```
Defines if this operation is transpose-safe, which means that the result of op(input) is equivalent to op(t(input)). Usually, this applies to aggregate operations with fixed output dimension. Finally, this information is very useful in order to safely optimize the plan for sparse vectors, which otherwise would be (currently) always represented dense.

Returns:

always returns false

checkAndSetForcedPlatform

public void checkAndSetForcedPlatform()

checkAndSetInvalidCPDimsAndSize

public void checkAndSetInvalidCPDimsAndSize()

hasValidCPDimsAndSize

public boolean hasValidCPDimsAndSize()

hasMatrixInputWithDifferentBlocksizes

public boolean hasMatrixInputWithDifferentBlocksizes()

setRequiresReblock

public void setRequiresReblock(boolean flag)

requiresReblock
```
public boolean requiresReblock()
```

setRequiresCheckpoint

public void setRequiresCheckpoint(boolean flag)

requiresCheckpoint
```
public boolean requiresCheckpoint()
```

setRequiresCompression
```
public void setRequiresCompression()
```

setRequiresCompression

public void setRequiresCompression(AWTreeNode node)

setRequiresDeCompression

public void setRequiresDeCompression()

isRequiredDecompression

public boolean isRequiredDecompression()

requiresCompression
```
public boolean requiresCompression()
```

setCompressedOutput

public void setCompressedOutput(boolean value)

setCompressedSize

public void setCompressedSize(long size)

getCompressedSize
```
public long getCompressedSize()
```

isCompressedOutput
```
public boolean isCompressedOutput()
```

hasCompressedInput
```
public boolean hasCompressedInput()
```

compressedSize
```
public long compressedSize()
```

setRequiresLineageCaching

public void setRequiresLineageCaching(boolean flag)

requiresLineageCaching

public boolean requiresLineageCaching()

constructAndSetLopsDataFlowProperties

public void constructAndSetLopsDataFlowProperties()

createOffsetLop

public static Lop createOffsetLop(Hop hop,
                                  boolean repCols)

setOutputEmptyBlocks

public void setOutputEmptyBlocks(boolean flag)

isOutputEmptyBlocks
```
public boolean isOutputEmptyBlocks()
```

getInputOutputSize

public double getInputOutputSize(Collection<String> exclVars)

getMemEstimate
```
public double getMemEstimate()
```
NOTES: * Purpose: Whenever the output dimensions / sparsity of a hop are unknown, this hop should store its worst-case output statistics (if known) in that table. Subsequent hops can then * Invocation: Intended to be called for ALL root nodes of one Hops DAG with the same (initially empty) memo table.

Returns:

memory estimate

setMemEstimate
```
public void setMemEstimate(double mem)
```
Sets memory estimate in bytes

Parameters:

mem - memory estimate

clearMemEstimate
```
public void clearMemEstimate()
```

isMemEstimated
```
public boolean isMemEstimated()
```

getInputMemEstimate
```
public double getInputMemEstimate()
```

getOutputMemEstimate
```
public double getOutputMemEstimate()
```

getIntermediateMemEstimate

public double getIntermediateMemEstimate()

getSpBroadcastSize
```
public double getSpBroadcastSize()
```

computeMemEstimate
```
public void computeMemEstimate(MemoTable memo)
```
Computes the estimate of memory required to store the input/output of this hop in memory. This is the default implementation (orchestration of hop-specific implementation) that should suffice for most hops. If a hop requires more control, this method should be overwritten with awareness of (1) output estimates, and (2) propagation of worst-case matrix characteristics (dimensions, sparsity). TODO remove memo table and, on constructor refresh, inference in refresh, single compute mem, maybe general computeMemEstimate, flags to indicate if estimate or not.

Parameters:

memo - memory table

refreshMemEstimates
```
public void refreshMemEstimates(MemoTable memo)
```
Recursively computes memory estimates for all the Hops in the DAG rooted at the current hop pointed by this.

Parameters:

memo - memory table

isFederated
```
public boolean isFederated()
```

isFederatedOutput
```
public boolean isFederatedOutput()
```

someInputFederated
```
public boolean someInputFederated()
```

getParent
```
public ArrayList<Hop> getParent()
```

getInput
```
public ArrayList<Hop> getInput()
```

getInput
```
public Hop getInput(int ix)
```

addInput
```
public void addInput(Hop h)
```

addAllInputs

public void addAllInputs(ArrayList<Hop> list)

getBlocksize
```
public int getBlocksize()
```

setBlocksize
```
public void setBlocksize(int blen)
```

setNnz
```
public void setNnz(long nnz)
```

getNnz
```
public long getNnz()
```

setPrivacy

public void setPrivacy(PrivacyConstraint privacy)

getPrivacy
```
public PrivacyConstraint getPrivacy()
```

hasFederatedOutput
```
public boolean hasFederatedOutput()
```

setUpdateType

public void setUpdateType(MatrixObject.UpdateType update)

getUpdateType

public MatrixObject.UpdateType getUpdateType()

constructLops
```
public abstract Lop constructLops()
```

getOpString
```
public abstract String getOpString()
```

toString
```
public String toString()
```
Overrides:

toString in class Object

isGPUEnabled
```
public abstract boolean isGPUEnabled()
```
In memory-based optimizer mode (see OptimizerUtils.isMemoryBasedOptLevel()), the exectype is determined by checking this method as well as memory budget of this Hop. Please see findExecTypeByMemEstimate for more detail. This method is necessary because not all operator are supported efficiently on GPU (for example: operations on frames and scalar as well as operations such as table).

Returns:

true if the Hop is eligible for GPU Exectype.

dimsKnown
```
public boolean dimsKnown()
```

dimsKnown

public boolean dimsKnown(boolean includeNnz)

dimsKnownAny
```
public boolean dimsKnownAny()
```

rowsKnown
```
public boolean rowsKnown()
```

colsKnown
```
public boolean colsKnown()
```

resetVisitStatus

public static void resetVisitStatus(ArrayList<Hop> hops)

resetVisitStatus

public static void resetVisitStatus(ArrayList<Hop> hops,
                                    boolean force)

resetVisitStatus
```
public Hop resetVisitStatus()
```

resetVisitStatusForced

public void resetVisitStatusForced(HashSet<Long> memo)

resetRecompilationFlag

public static void resetRecompilationFlag(ArrayList<Hop> hops,
                                          Types.ExecType et,
                                          Recompiler.ResetType reset)

resetRecompilationFlag

public static void resetRecompilationFlag(Hop hops,
                                          Types.ExecType et,
                                          Recompiler.ResetType reset)

getDim1
```
public long getDim1()
```

setDim1
```
public void setDim1(long dim1)
```

getDim2
```
public long getDim2()
```

setDim2
```
public void setDim2(long dim2)
```

getDim
```
public long getDim(int i)
```

setDim

public void setDim(int i,
                   long dim)

getLength
```
public long getLength()
```

getSparsity
```
public double getSparsity()
```

getDataCharacteristics

public DataCharacteristics getDataCharacteristics()

getLops
```
public Lop getLops()
```

setLops
```
public void setLops(Lop lops)
```

isVisited
```
public boolean isVisited()
```

getDataType
```
public Types.DataType getDataType()
```

setDataType

public void setDataType(Types.DataType dt)

isScalar
```
public boolean isScalar()
```

isMatrix
```
public boolean isMatrix()
```

setVisited
```
public void setVisited()
```

setVisited
```
public void setVisited(boolean flag)
```

setName
```
public void setName(String _name)
```

getName
```
public String getName()
```

getValueType
```
public Types.ValueType getValueType()
```

setValueType

public void setValueType(Types.ValueType vt)

requiresRecompile
```
public boolean requiresRecompile()
```
Indicates if dynamic recompilation is required for this hop.

Returns:

true if dynamic recompilation required

setRequiresRecompile
```
public void setRequiresRecompile()
```
Marks the hop for dynamic recompilation.

refreshSizeInformation
```
public abstract void refreshSizeInformation()
```
Update the output size information for this hop.

computeSizeInformation

public static long computeSizeInformation(Hop input)

refreshRowsParameterInformation

public void refreshRowsParameterInformation(Hop input,
                                            LocalVariableMap vars)

refreshRowsParameterInformation

public void refreshRowsParameterInformation(Hop input,
                                            LocalVariableMap vars,
                                            HashMap<Long,Long> memo)

refreshColsParameterInformation

public void refreshColsParameterInformation(Hop input,
                                            LocalVariableMap vars)

refreshColsParameterInformation

public void refreshColsParameterInformation(Hop input,
                                            LocalVariableMap vars,
                                            HashMap<Long,Long> memo)

computeSizeInformation

public long computeSizeInformation(Hop input,
                                   LocalVariableMap vars)

computeSizeInformation

public long computeSizeInformation(Hop input,
                                   LocalVariableMap vars,
                                   HashMap<Long,Long> memo)

computeBoundsInformation

public double computeBoundsInformation(Hop input)

computeBoundsInformation

public static double computeBoundsInformation(Hop input,
                                              LocalVariableMap vars)

computeBoundsInformation

public static double computeBoundsInformation(Hop input,
                                              LocalVariableMap vars,
                                              HashMap<Long,Double> memo)

clone

public abstract Object clone()
                      throws CloneNotSupportedException

Overrides:: clone in class Object
Throws:: CloneNotSupportedException

compare

public abstract boolean compare(Hop that)

setBeginLine
```
public void setBeginLine(int passed)
```
Specified by:

setBeginLine in interface ParseInfo

setBeginColumn
```
public void setBeginColumn(int passed)
```
Specified by:

setBeginColumn in interface ParseInfo

setEndLine
```
public void setEndLine(int passed)
```
Specified by:

setEndLine in interface ParseInfo

setEndColumn
```
public void setEndColumn(int passed)
```
Specified by:

setEndColumn in interface ParseInfo

setFilename
```
public void setFilename(String passed)
```
Specified by:

setFilename in interface ParseInfo

setText
```
public void setText(String text)
```
Specified by:

setText in interface ParseInfo

getBeginLine
```
public int getBeginLine()
```
Specified by:

getBeginLine in interface ParseInfo

getBeginColumn
```
public int getBeginColumn()
```
Specified by:

getBeginColumn in interface ParseInfo

getEndLine
```
public int getEndLine()
```
Specified by:

getEndLine in interface ParseInfo

getEndColumn
```
public int getEndColumn()
```
Specified by:

getEndColumn in interface ParseInfo

getFilename
```
public String getFilename()
```
Specified by:

getFilename in interface ParseInfo

getText
```
public String getText()
```
Specified by:

getText in interface ParseInfo

printErrorLocation
```
public String printErrorLocation()
```

setParseInfo
```
public void setParseInfo(ParseInfo parseInfo)
```
Set parse information.

Parameters:

parseInfo - parse information, such as beginning line position, beginning column position, ending line position, ending column position, text, and filename

Class Hop

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Methods inherited from interface org.apache.sysds.parser.ParseInfo

Field Detail

CPThreshold

_beginLine

_beginColumn

_endLine

_endColumn

_filename

_text

Constructor Detail

Hop

Method Detail

getHopID

checkArity

getExecType

setExecType

setFederatedOutput

resetExecType

getForcedExecType

setForcedExecType

allowsAllExecTypes

isTransposeSafe

checkAndSetForcedPlatform

checkAndSetInvalidCPDimsAndSize

hasValidCPDimsAndSize

hasMatrixInputWithDifferentBlocksizes

setRequiresReblock

requiresReblock

setRequiresCheckpoint

requiresCheckpoint

setRequiresCompression

setRequiresCompression

setRequiresDeCompression

isRequiredDecompression

requiresCompression

setCompressedOutput

setCompressedSize

getCompressedSize

isCompressedOutput

hasCompressedInput

compressedSize

setRequiresLineageCaching

requiresLineageCaching

constructAndSetLopsDataFlowProperties

createOffsetLop

setOutputEmptyBlocks

isOutputEmptyBlocks

getInputOutputSize

getMemEstimate

setMemEstimate

clearMemEstimate

isMemEstimated

getInputMemEstimate

getOutputMemEstimate

getIntermediateMemEstimate

getSpBroadcastSize

computeMemEstimate

refreshMemEstimates

isFederated

isFederatedOutput

someInputFederated

getParent

getInput

getInput

addInput

addAllInputs

getBlocksize

setBlocksize

setNnz

getNnz

setPrivacy

getPrivacy

hasFederatedOutput

setUpdateType

getUpdateType