Package org.apache.sysds.runtime.instructions.spark.data

Class PartitionedBlock<T extends CacheBlock>

  • All Implemented Interfaces:
    Externalizable, Serializable
    public class PartitionedBlock<T extends CacheBlock>
extends Object
implements Externalizable
    This class is for partitioned matrix/frame blocks, to be used as broadcasts. Distributed tasks require block-partitioned broadcasts but a lazy partitioning per task would create instance-local copies and hence replicate broadcast variables which are shared by all tasks within an executor.
    See Also:
    Serialized Form

    • Constructor Detail

      • PartitionedBlock

        public PartitionedBlock()

      • PartitionedBlock

        public PartitionedBlock​(T block,
                        int blen)

      • PartitionedBlock

        public PartitionedBlock​(T block,
                        long[] dims,
                        int blen)

      • PartitionedBlock

        public PartitionedBlock​(int rlen,
                        int clen,
                        int blen)

    • Method Detail

      • createPartition

        public PartitionedBlock<T> createPartition​(int offset,
                                           int numBlks)

      • getNumRows

        public long getNumRows()

      • getNumCols

        public long getNumCols()

      • getDim

        public long getDim​(int i)

      • getBlocksize

        public long getBlocksize()

      • getNumRowBlocks

        public int getNumRowBlocks()

      • getNumColumnBlocks

        public int getNumColumnBlocks()

      • getNumDimBlocks

        public int getNumDimBlocks​(int dim)

      • getBlock

        public T getBlock​(int rowIndex,
                  int colIndex)

      • getBlock

        public T getBlock​(int[] ix)

      • setBlock

        public void setBlock​(int rowIndex,
                     int colIndex,
                     T block)

      • getInMemorySize

        public long getInMemorySize()

      • getExactSerializedSize

        public long getExactSerializedSize()

      • clearBlocks

        public void clearBlocks()

      • readExternal

        public void readExternal​(ObjectInput is)
                  throws IOException
        Redirects the default java serialization via externalizable to our default hadoop writable serialization for efficient broadcast deserialization.
        Specified by:
        readExternal in interface Externalizable
        Parameters:
        is - object input
        Throws:
        IOException - if IOException occurs

      • writeExternal

        public void writeExternal​(ObjectOutput os)
                   throws IOException
        Redirects the default java serialization via externalizable to our default hadoop writable serialization for efficient broadcast serialization.
        Specified by:
        writeExternal in interface Externalizable
        Parameters:
        os - object output
        Throws:
        IOException - if IOException occurs