org.apache.sysds.runtime.io

Class ReaderWriterFederated

java.lang.Object

org.apache.sysds.runtime.io.ReaderWriterFederated

public class ReaderWriterFederated
extends Object

This class serves as the reader for federated objects. To read the files a mdt file is required. The reader is different from the other readers in the since that it does not return a MatrixBlock but a Matrix Object wrapper, containing the federated Mapping. On the Matrix Object the function isFederated() will if called read in the federated locations and instantiate the map. The reading is done through this code. This means in practice that it circumvent the other reading code. See more in: org.apache.sysds.runtime.controlprogram.caching.MatrixObject.readBlobFromHDFS() org.apache.sysds.runtime.controlprogram.caching.CacheableData.isFederated()

Constructor Summary

Constructors

Constructor and Description
ReaderWriterFederated()

Method Summary

Modifier and Type	Method and Description
static List<org.apache.commons.lang3.tuple.Pair<FederatedRange,FederatedData>>	read(String file, DataCharacteristics mc) Read a federated map from disk, It is not initialized before it is used in: org.apache.sysds.runtime.instructions.fed.InitFEDInstruction
static void	write(String file, FederationMap fedMap) TODO add writing to each of the federated locations so that they also save their matrices.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ReaderWriterFederated

public ReaderWriterFederated()

Method Detail

read

public static List<org.apache.commons.lang3.tuple.Pair<FederatedRange,FederatedData>> read(String file,
                                                                                           DataCharacteristics mc)

Read a federated map from disk, It is not initialized before it is used in: org.apache.sysds.runtime.instructions.fed.InitFEDInstruction

Parameters:

file - The file to read (defaults to HDFS)

mc - The data characteristics of the file, that can be read from the mtd file.

Returns:

A List of federatedRanges and Federated Data

write

public static void write(String file,
                         FederationMap fedMap)

TODO add writing to each of the federated locations so that they also save their matrices. Currently this would write the federated matrix to disk only locally.

Parameters:

file - The file to save to, (defaults to HDFS paths)

fedMap - The federated map to save.