Package org.apache.sysds.runtime.matrix.data

Class LibMatrixReorg

java.lang.Object

org.apache.sysds.runtime.matrix.data.LibMatrixReorg

public class LibMatrixReorg
extends Object

MB: Library for selected matrix reorg operations including special cases and all combinations of dense and sparse representations. Current list of supported operations: - reshape, - r' (transpose), - rdiag (diagV2M/diagM2V), - rsort (sorting data/indexes) - rmempty (remove empty) - rexpand (outer/table-seq expansion)

Field Summary

Fields

Modifier and Type	Field	Description
static long	PAR_NUMCELL_THRESHOLD
static int	PAR_NUMCELL_THRESHOLD_SORT
static boolean	SHALLOW_COPY_REORG
static boolean	SPARSE_OUTPUTS_IN_CSR

Method Summary

Modifier and Type	Method	Description
static void	checkRexpand(MatrixBlock in, boolean ignore)	Quick check if the input is valid for rexpand, this check does not guarantee that the input is valid for rexpand
static int[]	countNNZColumns(MatrixBlock in, int k, ExecutorService pool)
static List<Future<int[]>>	countNNZColumnsFuture(MatrixBlock in, int k, ExecutorService pool)
static int[]	countNnzPerColumn(MatrixBlock in)
static int[]	countNnzPerColumn(MatrixBlock in, int rl, int ru)
static MatrixBlock	diag(MatrixBlock in, MatrixBlock out)
static boolean	isSupportedReorgOperator(ReorgOperator op)
static int[]	mergeNnzCounts(int[] cnt, int[] cnt2)
static MatrixBlock	reorg(MatrixBlock in, MatrixBlock out, ReorgOperator op)
static MatrixBlock	reorgInPlace(MatrixBlock in, ReorgOperator op)
static List<IndexedMatrixValue>	reshape(IndexedMatrixValue in, DataCharacteristics mcIn, DataCharacteristics mcOut, boolean rowwise, boolean outputEmptyBlocks)	MR/SPARK reshape interface - for reshape we cannot view blocks independently, and hence, there are different CP and MR interfaces.
static MatrixBlock	reshape(MatrixBlock in, MatrixBlock out, int rows, int cols, boolean rowwise)	CP reshape operation (single input, single output matrix) NOTE: In contrast to R, the rowwise parameter specifies both the read and write order, with row-wise being the default, while R uses always a column-wise read, rowwise specifying the write order and column-wise being the default.
static void	rev(IndexedMatrixValue in, long rlen, int blen, ArrayList<IndexedMatrixValue> out)
static MatrixBlock	rev(MatrixBlock in, MatrixBlock out)
static void	rexpand(IndexedMatrixValue data, double max, boolean rows, boolean cast, boolean ignore, long blen, ArrayList<IndexedMatrixValue> outList)	MR/Spark rexpand operation (single input, multiple outputs incl empty blocks)
static MatrixBlock	rexpand(MatrixBlock in, MatrixBlock ret, double max, boolean rows, boolean cast, boolean ignore, int k)	CP rexpand operation (single input, single output), the classic example of this operation is one hot encoding of a column to multiple columns.
static MatrixBlock	rexpand(MatrixBlock in, MatrixBlock ret, int max, boolean rows, boolean cast, boolean ignore, int k)	CP rexpand operation (single input, single output), the classic example of this operation is one hot encoding of a column to multiple columns.
static void	rmempty(IndexedMatrixValue data, IndexedMatrixValue offset, boolean rmRows, long len, long blen, ArrayList<IndexedMatrixValue> outList)	MR rmempty interface - for rmempty we cannot view blocks independently, and hence, there are different CP and MR interfaces.
static MatrixBlock	rmempty(MatrixBlock in, MatrixBlock ret, boolean rows, boolean emptyReturn, MatrixBlock select)	CP rmempty operation (single input, single output matrix)
static MatrixBlock	sort(MatrixBlock in, MatrixBlock out, int[] by, boolean desc, boolean ixret)
static MatrixBlock	sort(MatrixBlock in, MatrixBlock out, int[] by, boolean desc, boolean ixret, int k)
static MatrixBlock	transpose(MatrixBlock in)
static MatrixBlock	transpose(MatrixBlock in, int k)
static MatrixBlock	transpose(MatrixBlock in, MatrixBlock out)
static MatrixBlock	transpose(MatrixBlock in, MatrixBlock out, int k)
static MatrixBlock	transpose(MatrixBlock in, MatrixBlock out, int k, boolean allowCSR)
static MatrixBlock	transposeInPlace(MatrixBlock in, int k)

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PAR_NUMCELL_THRESHOLD

public static long PAR_NUMCELL_THRESHOLD

PAR_NUMCELL_THRESHOLD_SORT

public static final int PAR_NUMCELL_THRESHOLD_SORT

See Also:

Constant Field Values

SHALLOW_COPY_REORG

public static final boolean SHALLOW_COPY_REORG

See Also:

Constant Field Values

SPARSE_OUTPUTS_IN_CSR

public static final boolean SPARSE_OUTPUTS_IN_CSR

See Also:

Constant Field Values

Method Detail

isSupportedReorgOperator

public static boolean isSupportedReorgOperator(ReorgOperator op)

reorg

public static MatrixBlock reorg(MatrixBlock in,
                                MatrixBlock out,
                                ReorgOperator op)

reorgInPlace

public static MatrixBlock reorgInPlace(MatrixBlock in,
                                       ReorgOperator op)

transpose

public static MatrixBlock transpose(MatrixBlock in)

transpose

public static MatrixBlock transpose(MatrixBlock in,
                                    MatrixBlock out)

transpose

public static MatrixBlock transpose(MatrixBlock in,
                                    int k)

transpose

public static MatrixBlock transpose(MatrixBlock in,
                                    MatrixBlock out,
                                    int k)

transpose

public static MatrixBlock transpose(MatrixBlock in,
                                    MatrixBlock out,
                                    int k,
                                    boolean allowCSR)

countNNZColumns

public static int[] countNNZColumns(MatrixBlock in,
                                    int k,
                                    ExecutorService pool)

countNNZColumnsFuture

public static List<Future<int[]>> countNNZColumnsFuture(MatrixBlock in,
                                                        int k,
                                                        ExecutorService pool)
                                                 throws InterruptedException

Throws:

InterruptedException

transposeInPlace

public static MatrixBlock transposeInPlace(MatrixBlock in,
                                           int k)

rev

public static MatrixBlock rev(MatrixBlock in,
                              MatrixBlock out)

rev

public static void rev(IndexedMatrixValue in,
                       long rlen,
                       int blen,
                       ArrayList<IndexedMatrixValue> out)

diag

public static MatrixBlock diag(MatrixBlock in,
                               MatrixBlock out)

sort

public static MatrixBlock sort(MatrixBlock in,
                               MatrixBlock out,
                               int[] by,
                               boolean desc,
                               boolean ixret)

sort

public static MatrixBlock sort(MatrixBlock in,
                               MatrixBlock out,
                               int[] by,
                               boolean desc,
                               boolean ixret,
                               int k)

Parameters:

in - Input matrix to sort

out - Output matrix where the sorted input is inserted to

by - The Ordering parameter

desc - A boolean, specifying if it should be descending order.

ixret - A boolean, specifying if the return should be the sorted indexes.

k - Number of parallel threads

Returns:

The sorted out matrix.

reshape

public static MatrixBlock reshape(MatrixBlock in,
                                  MatrixBlock out,
                                  int rows,
                                  int cols,
                                  boolean rowwise)

CP reshape operation (single input, single output matrix) NOTE: In contrast to R, the rowwise parameter specifies both the read and write order, with row-wise being the default, while R uses always a column-wise read, rowwise specifying the write order and column-wise being the default.

Parameters:

in - input matrix

out - output matrix

rows - number of rows

cols - number of columns

rowwise - if true, reshape by row

Returns:

output matrix

reshape

public static List<IndexedMatrixValue> reshape(IndexedMatrixValue in,
                                               DataCharacteristics mcIn,
                                               DataCharacteristics mcOut,
                                               boolean rowwise,
                                               boolean outputEmptyBlocks)

MR/SPARK reshape interface - for reshape we cannot view blocks independently, and hence, there are different CP and MR interfaces.

Parameters:

in - indexed matrix value

mcIn - input matrix characteristics

mcOut - output matrix characteristics

rowwise - if true, reshape by row

outputEmptyBlocks - output blocks with nnz=0

Returns:

list of indexed matrix values

rmempty

public static MatrixBlock rmempty(MatrixBlock in,
                                  MatrixBlock ret,
                                  boolean rows,
                                  boolean emptyReturn,
                                  MatrixBlock select)

CP rmempty operation (single input, single output matrix)

Parameters:

in - input matrix

ret - output matrix

rows - ?

emptyReturn - return row/column of zeros for empty input

select - ?

Returns:

matrix block

rmempty

public static void rmempty(IndexedMatrixValue data,
                           IndexedMatrixValue offset,
                           boolean rmRows,
                           long len,
                           long blen,
                           ArrayList<IndexedMatrixValue> outList)

MR rmempty interface - for rmempty we cannot view blocks independently, and hence, there are different CP and MR interfaces.

Parameters:

data - ?

offset - ?

rmRows - ?

len - ?

blen - block length

outList - list of indexed matrix values

rexpand

public static MatrixBlock rexpand(MatrixBlock in,
                                  MatrixBlock ret,
                                  double max,
                                  boolean rows,
                                  boolean cast,
                                  boolean ignore,
                                  int k)

CP rexpand operation (single input, single output), the classic example of this operation is one hot encoding of a column to multiple columns.

Parameters:

in - Input matrix

ret - Output matrix

max - Number of rows/cols of the output

rows - If the expansion is in rows direction

cast - If the values contained should be cast to double (rounded up and down)

ignore - Ignore if the input contain values below zero that technically is incorrect input.

k - Degree of parallelism

Returns:

Output matrix rexpanded

rexpand

public static MatrixBlock rexpand(MatrixBlock in,
                                  MatrixBlock ret,
                                  int max,
                                  boolean rows,
                                  boolean cast,
                                  boolean ignore,
                                  int k)

CP rexpand operation (single input, single output), the classic example of this operation is one hot encoding of a column to multiple columns.

Parameters:

in - Input matrix

ret - Output matrix

max - Number of rows/cols of the output

rows - If the expansion is in rows direction

cast - If the values contained should be cast to double (rounded up and down)

ignore - Ignore if the input contain values below zero that technically is incorrect input.

k - Degree of parallelism

Returns:

Output matrix rexpanded

checkRexpand

public static void checkRexpand(MatrixBlock in,
                                boolean ignore)

Quick check if the input is valid for rexpand, this check does not guarantee that the input is valid for rexpand

Parameters:

in - Input matrix block

ignore - If zero valued cells should be ignored

rexpand

public static void rexpand(IndexedMatrixValue data,
                           double max,
                           boolean rows,
                           boolean cast,
                           boolean ignore,
                           long blen,
                           ArrayList<IndexedMatrixValue> outList)

MR/Spark rexpand operation (single input, multiple outputs incl empty blocks)

Parameters:

data - Input indexed matrix block

max - Total nrows/cols of the output

rows - If the expansion is in rows direction

cast - If the values contained should be cast to double (rounded up and down)

ignore - Ignore if the input contain values below zero that technically is incorrect input.

blen - The block size to slice the output up into

outList - The output indexedMatrixValues (a list to add all the output blocks to / modify)

countNnzPerColumn

public static int[] countNnzPerColumn(MatrixBlock in)

countNnzPerColumn

public static int[] countNnzPerColumn(MatrixBlock in,
                                      int rl,
                                      int ru)

mergeNnzCounts

public static int[] mergeNnzCounts(int[] cnt,
                                   int[] cnt2)