Package org.apache.sysds.runtime.controlprogram.caching

Class MatrixObject

java.lang.Object

org.apache.sysds.runtime.instructions.cp.Data

org.apache.sysds.runtime.controlprogram.caching.CacheableData<MatrixBlock>

org.apache.sysds.runtime.controlprogram.caching.MatrixObject

All Implemented Interfaces:

Serializable

public class MatrixObject
extends CacheableData<MatrixBlock>

Represents a matrix in control program. This class contains method to read matrices from HDFS and convert them to a specific format/representation. It is also able to write several formats/representation of matrices to HDFS. IMPORTANT: Preserve one-to-one correspondence between MatrixObject and MatrixBlock objects, for cache purposes. Do not change a MatrixBlock object without informing its MatrixObject object.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	MatrixObject.UpdateType

Nested classes/interfaces inherited from class org.apache.sysds.runtime.controlprogram.caching.CacheableData

CacheableData.CacheStatus

Field Summary

Fields inherited from class org.apache.sysds.runtime.controlprogram.caching.CacheableData

cacheEvictionLocalFilePath, cacheEvictionLocalFilePrefix, CACHING_ASYNC_FILECLEANUP, CACHING_ASYNC_SERIALIZE, CACHING_BUFFER_PAGECACHE, CACHING_BUFFER_POLICY, CACHING_COUNTER_GROUP_NAME, CACHING_EVICTION_FILEEXTENSION, CACHING_THRESHOLD, CACHING_WRITE_CACHE_ON_READ

Constructor Summary

Constructors

Constructor	Description
MatrixObject(Types.ValueType vt, String file)	Constructor that takes the value type and the HDFS filename.
MatrixObject(Types.ValueType vt, String file, MetaData mtd)	Constructor that takes the value type, HDFS filename and associated metadata.
MatrixObject(Types.ValueType vt, String file, MetaData mtd, MatrixBlock data)	Constructor that takes the value type, HDFS filename and associated metadata and a MatrixBlock used for creation after serialization
MatrixObject(MatrixObject mo)	Copy constructor that copies meta data but NO data.

Method Summary

Modifier and Type	Method	Description
long	getBlocksize()
long	getNnz()
String	getPartitionFileName(IndexRange pred, int blen)
ParForProgramBlock.PDataPartitionFormat	getPartitionFormat()
int	getPartitionSize()
double	getSparsity()
MatrixObject.UpdateType	getUpdateType()
boolean	isDiag()
boolean	isMarked()
boolean	isPartitioned()
MatrixBlock	readMatrixPartition(IndexRange pred)	NOTE: for reading matrix partitions, we could cache (in its real sense) the read block with soft references (no need for eviction, as partitioning only applied for read-only matrices).
void	refreshMetaData()	Make the matrix metadata consistent with the in-memory matrix data
void	setDiag(boolean diag)
void	setInMemoryPartition(MatrixBlock block)
void	setMarkForLinCache(boolean mark)
void	setPartitioned(ParForProgramBlock.PDataPartitionFormat format, int n)
void	setUpdateType(MatrixObject.UpdateType flag)
String	toString()
void	unsetPartitioned()
void	updateDataCharacteristics(DataCharacteristics dc)

Methods inherited from class org.apache.sysds.runtime.controlprogram.caching.CacheableData

acquireModify, acquireRead, acquireReadAndRelease, addBroadcastSize, cleanupCacheDir, cleanupCacheDir, clearData, clearData, disableCaching, enableCaching, enableCleanup, exportData, exportData, exportData, exportData, exportData, freeEvictedBlob, getBroadcastHandle, getBroadcastSize, getCacheLineage, getCompressedSize, getDataCharacteristics, getDataSize, getDebugName, getDim, getFedMapping, getFileFormatProperties, getFileName, getGPUObject, getMetaData, getNumColumns, getNumRows, getRDDHandle, getStatus, getUniqueID, hasValidLineage, initCaching, initCaching, isBelowCachingThreshold, isCached, isCachingActive, isCleanupEnabled, isCompressed, isDirty, isFederated, isFederated, isFederatedExcept, isHDFSFileExists, isPendingRDDOps, moveData, release, removeGPUObject, removeMetaData, setBroadcastHandle, setCacheLineage, setCompressedSize, setDirty, setEmptyStatus, setFedMapping, setFileFormatProperties, setFileName, setGPUObject, setHDFSFileExists, setMetaData, setRDDHandle

Methods inherited from class org.apache.sysds.runtime.instructions.cp.Data

getDataType, getPrivacyConstraint, getValueType, setPrivacyConstraints

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

MatrixObject

public MatrixObject(Types.ValueType vt,
                    String file)

Constructor that takes the value type and the HDFS filename.

Parameters:

vt - value type

file - file name

MatrixObject

public MatrixObject(Types.ValueType vt,
                    String file,
                    MetaData mtd)

Constructor that takes the value type, HDFS filename and associated metadata.

Parameters:

vt - value type

file - file name

mtd - metadata

MatrixObject

public MatrixObject(Types.ValueType vt,
                    String file,
                    MetaData mtd,
                    MatrixBlock data)

Constructor that takes the value type, HDFS filename and associated metadata and a MatrixBlock used for creation after serialization

Parameters:

vt - value type

file - file name

mtd - metadata

data - matrix block data

MatrixObject

public MatrixObject(MatrixObject mo)

Copy constructor that copies meta data but NO data.

Parameters:

mo - matrix object

Method Detail

setUpdateType

public void setUpdateType(MatrixObject.UpdateType flag)

getUpdateType

public MatrixObject.UpdateType getUpdateType()

isDiag

public boolean isDiag()

setDiag

public void setDiag(boolean diag)

setMarkForLinCache

public void setMarkForLinCache(boolean mark)

isMarked

public boolean isMarked()

updateDataCharacteristics

public void updateDataCharacteristics(DataCharacteristics dc)

Overrides:

updateDataCharacteristics in class Data

refreshMetaData

public void refreshMetaData()

Make the matrix metadata consistent with the in-memory matrix data

Specified by:

refreshMetaData in class CacheableData<MatrixBlock>

getBlocksize

public long getBlocksize()

Overrides:

getBlocksize in class CacheableData<MatrixBlock>

getNnz

public long getNnz()

getSparsity

public double getSparsity()

setPartitioned

public void setPartitioned(ParForProgramBlock.PDataPartitionFormat format,
                           int n)

unsetPartitioned

public void unsetPartitioned()

isPartitioned

public boolean isPartitioned()

getPartitionFormat

public ParForProgramBlock.PDataPartitionFormat getPartitionFormat()

getPartitionSize

public int getPartitionSize()

setInMemoryPartition

public void setInMemoryPartition(MatrixBlock block)

readMatrixPartition

public MatrixBlock readMatrixPartition(IndexRange pred)

NOTE: for reading matrix partitions, we could cache (in its real sense) the read block with soft references (no need for eviction, as partitioning only applied for read-only matrices). However, since we currently only support row- and column-wise partitioning caching is not applied yet. This could be changed once we also support column-block-wise and row-block-wise. Furthermore, as we reject to partition vectors and support only full row or column indexing, no metadata (apart from the partition flag) is required.

Parameters:

pred - index range

Returns:

matrix block

getPartitionFileName

public String getPartitionFileName(IndexRange pred,
                                   int blen)

toString

public String toString()

Overrides:

toString in class CacheableData<MatrixBlock>