Package org.apache.sysds.hops

Class Hop

    • Field Detail

      • _beginLine

        public int _beginLine

      • _beginColumn

        public int _beginColumn

      • _endLine

        public int _endLine

      • _endColumn

        public int _endColumn

      • _filename

        public String _filename

    • Method Detail

      • getHopID

        public long getHopID()

      • checkArity

        public abstract void checkArity()
        Check whether this Hop has a correct number of inputs. (Some Hops can have a variable number of inputs, such as DataOp, DataGenOp, ParameterizedBuiltinOp, ReorgOp, TernaryOp, QuaternaryOp, MultipleOp, DnnOp, and SpoofFusedOp.) Parameterized Hops (such as DataOp) can check that the number of parameters matches the number of inputs.

      • activatePrefetch

        public void activatePrefetch()
        Activate prefetch of HOP.

      • deactivatePrefetch

        public void deactivatePrefetch()

      • prefetchActivated

        public boolean prefetchActivated()
        Checks if prefetch is activated for this hop.
        Returns:
        true if prefetch is activated

      • resetExecType

        public void resetExecType()

      • setForcedExecType

        public void setForcedExecType​(Types.ExecType etype)

      • allowsAllExecTypes

        public abstract boolean allowsAllExecTypes()

      • isTransposeSafe

        public boolean isTransposeSafe()
        Defines if this operation is transpose-safe, which means that the result of op(input) is equivalent to op(t(input)). Usually, this applies to aggregate operations with fixed output dimension. Finally, this information is very useful in order to safely optimize the plan for sparse vectors, which otherwise would be (currently) always represented dense.
        Returns:
        always returns false

      • checkAndSetForcedPlatform

        public void checkAndSetForcedPlatform()

      • checkAndSetInvalidCPDimsAndSize

        public void checkAndSetInvalidCPDimsAndSize()

      • hasValidCPDimsAndSize

        public boolean hasValidCPDimsAndSize()

      • hasMatrixInputWithDifferentBlocksizes

        public boolean hasMatrixInputWithDifferentBlocksizes()

      • setRequiresReblock

        public void setRequiresReblock​(boolean flag)

      • requiresReblock

        public boolean requiresReblock()

      • setRequiresCheckpoint

        public void setRequiresCheckpoint​(boolean flag)

      • requiresCheckpoint

        public boolean requiresCheckpoint()

      • setRequiresCompression

        public void setRequiresCompression()

      • setRequiresCompression

        public void setRequiresCompression​(AWTreeNode node)

      • setRequiresDeCompression

        public void setRequiresDeCompression()

      • isRequiredDecompression

        public boolean isRequiredDecompression()

      • requiresCompression

        public boolean requiresCompression()

      • setCompressedOutput

        public void setCompressedOutput​(boolean value)

      • setCompressedSize

        public void setCompressedSize​(long size)

      • getCompressedSize

        public long getCompressedSize()

      • isCompressedOutput

        public boolean isCompressedOutput()

      • hasCompressedInput

        public boolean hasCompressedInput()

      • compressedSize

        public long compressedSize()

      • setRequiresLineageCaching

        public void setRequiresLineageCaching​(boolean flag)

      • requiresLineageCaching

        public boolean requiresLineageCaching()

      • updateLopFedOut

        public void updateLopFedOut​(Lop lop)

      • constructAndSetLopsDataFlowProperties

        public void constructAndSetLopsDataFlowProperties()

      • createOffsetLop

        public static Lop createOffsetLop​(Hop hop,
                                  boolean repCols)

      • setOutputEmptyBlocks

        public void setOutputEmptyBlocks​(boolean flag)

      • isOutputEmptyBlocks

        public boolean isOutputEmptyBlocks()

      • getInputOutputSize

        public double getInputOutputSize​(Collection<String> exclVars)

      • getMemEstimate

        public double getMemEstimate()
        NOTES: * Purpose: Whenever the output dimensions / sparsity of a hop are unknown, this hop should store its worst-case output statistics (if known) in that table. * Invocation: Intended to be called for ALL root nodes of one Hops DAG with the same (initially empty) memo table.
        Returns:
        memory estimate in bytes

      • setMemEstimate

        public void setMemEstimate​(double mem)
        Sets memory estimate in bytes
        Parameters:
        mem - memory estimate

      • clearMemEstimate

        public void clearMemEstimate()

      • isMemEstimated

        public boolean isMemEstimated()

      • getInputMemEstimate

        public double getInputMemEstimate()
        Get the memory estimate of inputs as the sum of input estimates in bytes.
        Returns:
        input memory estimate in bytes

      • getInputMemEstimate

        public double getInputMemEstimate​(double injectedDefault)
        Get the memory estimate of inputs as the sum of input estimates in bytes.
        Parameters:
        injectedDefault - default memory estimate (bytes) used when the memory estimate of the input is negative
        Returns:
        input memory estimate in bytes

      • getOutputMemEstimate

        public double getOutputMemEstimate()
        Output memory estimate in bytes.
        Returns:
        output memory estimate in bytes

      • getOutputMemEstimate

        public double getOutputMemEstimate​(double injectedDefault)
        Output memory estimate in bytes with negative memory estimates replaced by the injected default. The injected default represents the memory estimate per output cell, hence it is multiplied by the estimated dimensions of the output of the hop.
        Parameters:
        injectedDefault - memory estimate to be returned in case the memory estimate defaults to a negative number
        Returns:
        output memory estimate in bytes

      • getIntermediateMemEstimate

        public double getIntermediateMemEstimate()

      • getSpBroadcastSize

        public double getSpBroadcastSize()

      • computeMemEstimate

        public void computeMemEstimate​(MemoTable memo)
        Computes the estimate of memory required to store the input/output of this hop in memory. This is the default implementation (orchestration of hop-specific implementation) that should suffice for most hops. If a hop requires more control, this method should be overwritten with awareness of (1) output estimates, and (2) propagation of worst-case matrix characteristics (dimensions, sparsity). TODO remove memo table and, on constructor refresh, inference in refresh, single compute mem, maybe general computeMemEstimate, flags to indicate if estimate or not.
        Parameters:
        memo - memory table

      • refreshMemEstimates

        public void refreshMemEstimates​(MemoTable memo)
        Recursively computes memory estimates for all the Hops in the DAG rooted at the current hop pointed by this.
        Parameters:
        memo - memory table

      • isFederated

        public boolean isFederated()
        Checks if ExecType is federated.
        Returns:
        true if ExecType is federated

      • someInputFederated

        public boolean someInputFederated()

      • isFederatedDataOp

        public boolean isFederatedDataOp()
        Checks if the hop is a DataOp with federated data.
        Returns:
        true if hop is a federated DataOp

      • getInput

        public Hop getInput​(int ix)

      • addInput

        public void addInput​(Hop h)

      • addAllInputs

        public void addAllInputs​(ArrayList<Hop> list)

      • getBlocksize

        public int getBlocksize()

      • setBlocksize

        public void setBlocksize​(int blen)

      • setNnz

        public void setNnz​(long nnz)

      • getNnz

        public long getNnz()

      • hasFederatedOutput

        public boolean hasFederatedOutput()

      • hasLocalOutput

        public boolean hasLocalOutput()

      • federatedCostInitialized

        public boolean federatedCostInitialized()
        Check if federated cost has been initialized for this Hop.
        Returns:
        true if federated cost has been initialized

      • setFederatedCost

        public void setFederatedCost​(FederatedCost cost)

      • resetFederatedCost

        public void resetFederatedCost()
        Reset federated cost of this hop and all children of this hop.

      • constructLops

        public abstract Lop constructLops()

      • getOpString

        public abstract String getOpString()

      • isGPUEnabled

        public abstract boolean isGPUEnabled()
        In memory-based optimizer mode (see OptimizerUtils.isMemoryBasedOptLevel()), the exectype is determined by checking this method as well as memory budget of this Hop. Please see findExecTypeByMemEstimate for more detail. This method is necessary because not all operator are supported efficiently on GPU (for example: operations on frames and scalar as well as operations such as table).
        Returns:
        true if the Hop is eligible for GPU Exectype.

      • dimsKnown

        public boolean dimsKnown()

      • dimsKnown

        public boolean dimsKnown​(boolean includeNnz)

      • dimsKnownAny

        public boolean dimsKnownAny()

      • rowsKnown

        public boolean rowsKnown()

      • colsKnown

        public boolean colsKnown()

      • resetVisitStatus

        public static void resetVisitStatus​(ArrayList<Hop> hops)

      • resetVisitStatus

        public static void resetVisitStatus​(ArrayList<Hop> hops,
                                    boolean force)

      • resetVisitStatus

        public Hop resetVisitStatus()

      • resetVisitStatusForced

        public void resetVisitStatusForced​(HashSet<Long> memo)

      • getDim1

        public long getDim1()
        Get the number of rows in the Hop.
        Returns:
        a long.

      • setDim1

        public void setDim1​(long dim1)

      • getDim2

        public long getDim2()
        Get the number of columns in the Hop.
        Returns:
        a long.

      • setDim2

        public void setDim2​(long dim2)

      • getDim

        public long getDim​(int i)

      • setDim

        public void setDim​(int i,
                   long dim)

      • getLength

        public long getLength()

      • getSparsity

        public double getSparsity()

      • getLops

        public Lop getLops()

      • setLops

        public void setLops​(Lop lops)

      • isVisited

        public boolean isVisited()

      • isScalar

        public boolean isScalar()

      • isMatrix

        public boolean isMatrix()

      • setVisited

        public void setVisited()

      • setVisited

        public void setVisited​(boolean flag)

      • setName

        public void setName​(String _name)

      • getName

        public String getName()

      • requiresRecompile

        public boolean requiresRecompile()
        Indicates if dynamic recompilation is required for this hop.
        Returns:
        true if dynamic recompilation required

      • setRequiresRecompile

        public void setRequiresRecompile()
        Marks the hop for dynamic recompilation.

      • refreshSizeInformation

        public abstract void refreshSizeInformation()
        Update the output size information for this hop.

      • computeSizeInformation

        public static long computeSizeInformation​(Hop input)

      • refreshRowsParameterInformation

        public void refreshRowsParameterInformation​(Hop input,
                                            LocalVariableMap vars)

      • refreshColsParameterInformation

        public void refreshColsParameterInformation​(Hop input,
                                            LocalVariableMap vars)

      • computeSizeInformation

        public long computeSizeInformation​(Hop input,
                                   LocalVariableMap vars)

      • computeBoundsInformation

        public double computeBoundsInformation​(Hop input)

      • computeBoundsInformation

        public static double computeBoundsInformation​(Hop input,
                                              LocalVariableMap vars)

      • updateRepetitionEstimates

        public void updateRepetitionEstimates​(double repetitions)

      • getRepetitions

        public double getRepetitions()

      • compare

        public abstract boolean compare​(Hop that)

      • setBeginLine

        public void setBeginLine​(int passed)
        Specified by:
        setBeginLine in interface ParseInfo

      • setBeginColumn

        public void setBeginColumn​(int passed)
        Specified by:
        setBeginColumn in interface ParseInfo

      • setEndLine

        public void setEndLine​(int passed)
        Specified by:
        setEndLine in interface ParseInfo

      • setEndColumn

        public void setEndColumn​(int passed)
        Specified by:
        setEndColumn in interface ParseInfo

      • printErrorLocation

        public String printErrorLocation()

      • setParseInfo

        public void setParseInfo​(ParseInfo parseInfo)
        Set parse information.
        Parameters:
        parseInfo - parse information, such as beginning line position, beginning column position, ending line position, ending column position, text, and filename