org.apache.sysds.runtime.controlprogram.federated

Class FederationMap

java.lang.Object

org.apache.sysds.runtime.controlprogram.federated.FederationMap

public class FederationMap
extends Object

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	FederationMap.FPartitioning
static class	FederationMap.FReplication
static class	FederationMap.FType

Constructor Summary

Constructors

Constructor and Description
FederationMap(List<org.apache.commons.lang3.tuple.Pair<FederatedRange,FederatedData>> fedMap)
FederationMap(long ID, List<org.apache.commons.lang3.tuple.Pair<FederatedRange,FederatedData>> fedMap)
FederationMap(long ID, List<org.apache.commons.lang3.tuple.Pair<FederatedRange,FederatedData>> fedMap, FederationMap.FType type)

Method Summary

Modifier and Type	Method and Description
FederationMap	bind(long rOffset, long cOffset, FederationMap that)
FederatedRequest	broadcast(CacheableData<?> data)
FederatedRequest	broadcast(ScalarObject scalar)
FederatedRequest[]	broadcastSliced(CacheableData<?> data, boolean transposed) Creates separate slices of an input data object according to the index ranges of federated data.
FederatedRequest[]	broadcastSliced(CacheableData<?> data, boolean isFrame, int[][] ix)
FederatedRequest	cleanup(long tid, long... id)
FederationMap	clone()
FederationMap	copyWithNewID() Copy the federation map with the next available federated ID as reference to the federated data.
FederationMap	copyWithNewID(long id) Copy the federation map with the given ID as reference to the federated data.
FederationMap	copyWithNewID(long id, long clen) Copy the federation map with the given ID as reference to the federated data and with given clen as end dimension for the columns in the range.
FederationMap	copyWithNewIDAndRange(long rowRangeEnd, long colRangeEnd, long outputID) Copy federated mapping while giving the federated data new IDs and setting the ranges from zero to row and column ends specified.
void	execCleanup(long tid, long... id)
Future<FederatedResponse>[]	execute(long tid, boolean wait, FederatedRange[] fedRange1, FederatedRequest elseFr, FederatedRequest[] frSlices1, FederatedRequest[] frSlices2, FederatedRequest... fr)
Future<FederatedResponse>[]	execute(long tid, boolean wait, FederatedRequest... fr)
Future<FederatedResponse>[]	execute(long tid, boolean wait, FederatedRequest[] frSlices, FederatedRequest... fr)
Future<FederatedResponse>[]	execute(long tid, boolean wait, FederatedRequest[] frSlices1, FederatedRequest[] frSlices2, FederatedRequest... fr)
Future<FederatedResponse>[]	execute(long tid, FederatedRequest... fr)
Future<FederatedResponse>[]	execute(long tid, FederatedRequest[] frSlices, FederatedRequest... fr)
Future<FederatedResponse>[]	executeMultipleSlices(long tid, boolean wait, FederatedRequest[][] frSlices, FederatedRequest[] fr)
FederationMap	filter(IndexRange ixrange)
void	forEachParallel(BiFunction<FederatedRange,FederatedData,Void> forEachFunction) Execute a function for each FederatedRange + FederatedData pair.
FederatedData[]	getFederatedData()
FederatedRange[]	getFederatedRanges()
long	getID()
List<org.apache.commons.lang3.tuple.Pair<FederatedRange,FederatedData>>	getMap()
long	getMaxIndexInRange(int dim)
int	getSize()
FederationMap.FType	getType()
FederationMap	identCopy(long tid, long id)
boolean	isAligned(FederationMap that, boolean transposed) Determines if the two federation maps are aligned row/column partitions at the same federated sites (which allows for purely federated operation)
boolean	isAligned(FederationMap that, boolean transposed, boolean equalRows, boolean equalCols) determines if the two federated data are aligned row/column partitions (depending on parameters equalRows/equalCols) at the same federated site (which often allows for purely federated operations)
boolean	isInitialized()
FederationMap	mapParallel(long newVarID, BiFunction<FederatedRange,FederatedData,Void> mappingFunction) Execute a function for each FederatedRange + FederatedData pair mapping the pairs to their new form by directly changing both FederatedRange and FederatedData.
List<org.apache.commons.lang3.tuple.Pair<FederatedRange,Future<FederatedResponse>>>	requestFederatedData()
void	reverseFedMap()
void	setType(FederationMap.FType type)
String	toString()
FederationMap	transpose()

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

FederationMap

public FederationMap(List<org.apache.commons.lang3.tuple.Pair<FederatedRange,FederatedData>> fedMap)

FederationMap

public FederationMap(long ID,
                     List<org.apache.commons.lang3.tuple.Pair<FederatedRange,FederatedData>> fedMap)

FederationMap

public FederationMap(long ID,
                     List<org.apache.commons.lang3.tuple.Pair<FederatedRange,FederatedData>> fedMap,
                     FederationMap.FType type)

Method Detail

getID

public long getID()

getType

public FederationMap.FType getType()

isInitialized

public boolean isInitialized()

setType

public void setType(FederationMap.FType type)

getSize

public int getSize()

getFederatedRanges

public FederatedRange[] getFederatedRanges()

getFederatedData

public FederatedData[] getFederatedData()

getMap

public List<org.apache.commons.lang3.tuple.Pair<FederatedRange,FederatedData>> getMap()

broadcast

public FederatedRequest broadcast(CacheableData<?> data)

broadcast

public FederatedRequest broadcast(ScalarObject scalar)

broadcastSliced

public FederatedRequest[] broadcastSliced(CacheableData<?> data,
                                          boolean transposed)

Creates separate slices of an input data object according to the index ranges of federated data. Theses slices are then wrapped in separate federated requests for broadcasting.

Parameters:

data - input data object (matrix, tensor, frame)

transposed - false: slice according to federated data, true: slice according to transposed federated data

Returns:

array of federated requests corresponding to federated data

broadcastSliced

public FederatedRequest[] broadcastSliced(CacheableData<?> data,
                                          boolean isFrame,
                                          int[][] ix)

isAligned

public boolean isAligned(FederationMap that,
                         boolean transposed)

Determines if the two federation maps are aligned row/column partitions at the same federated sites (which allows for purely federated operation)

Parameters:

that - FederationMap to check alignment with

transposed - true if that FederationMap should be transposed before checking alignment

Returns:

true if this and that FederationMap are aligned

isAligned

public boolean isAligned(FederationMap that,
                         boolean transposed,
                         boolean equalRows,
                         boolean equalCols)

determines if the two federated data are aligned row/column partitions (depending on parameters equalRows/equalCols) at the same federated site (which often allows for purely federated operations)

Parameters:

that - FederationMap to check alignment with

transposed - true if that FederationMap should be transposed before checking alignment

equalRows - true to indicate that the row dimension should be checked for alignment

equalCols - true to indicate that the col dimension should be checked for alignment

Returns:

true if this and that FederationMap are aligned

execute

public Future<FederatedResponse>[] execute(long tid,
                                           FederatedRequest... fr)

execute

public Future<FederatedResponse>[] execute(long tid,
                                           boolean wait,
                                           FederatedRequest... fr)

execute

public Future<FederatedResponse>[] execute(long tid,
                                           FederatedRequest[] frSlices,
                                           FederatedRequest... fr)

execute

public Future<FederatedResponse>[] execute(long tid,
                                           boolean wait,
                                           FederatedRequest[] frSlices,
                                           FederatedRequest... fr)

execute

public Future<FederatedResponse>[] execute(long tid,
                                           boolean wait,
                                           FederatedRange[] fedRange1,
                                           FederatedRequest elseFr,
                                           FederatedRequest[] frSlices1,
                                           FederatedRequest[] frSlices2,
                                           FederatedRequest... fr)

execute

public Future<FederatedResponse>[] execute(long tid,
                                           boolean wait,
                                           FederatedRequest[] frSlices1,
                                           FederatedRequest[] frSlices2,
                                           FederatedRequest... fr)

executeMultipleSlices

public Future<FederatedResponse>[] executeMultipleSlices(long tid,
                                                         boolean wait,
                                                         FederatedRequest[][] frSlices,
                                                         FederatedRequest[] fr)

requestFederatedData

public List<org.apache.commons.lang3.tuple.Pair<FederatedRange,Future<FederatedResponse>>> requestFederatedData()

cleanup

public FederatedRequest cleanup(long tid,
                                long... id)

execCleanup

public void execCleanup(long tid,
                        long... id)

identCopy

public FederationMap identCopy(long tid,
                               long id)

copyWithNewID

public FederationMap copyWithNewID()

Copy the federation map with the next available federated ID as reference to the federated data. This means that the federated map refers to the next federated data object on the workers.

Returns:

copied federation map with next federated ID

copyWithNewID

public FederationMap copyWithNewID(long id)

Copy the federation map with the given ID as reference to the federated data. This means that the federated map refers to the federated data object on the workers with the given ID.

Parameters:

id - federated data object ID

Returns:

copied federation map with given federated ID

copyWithNewID

public FederationMap copyWithNewID(long id,
                                   long clen)

Copy the federation map with the given ID as reference to the federated data and with given clen as end dimension for the columns in the range. This means that the federated map refers to the federated data object on the workers with the given ID.

Parameters:

id - federated data object ID

clen - column length of data objects on federated workers

Returns:

copied federation map with given federated ID and ranges adapted according to clen

copyWithNewIDAndRange

public FederationMap copyWithNewIDAndRange(long rowRangeEnd,
                                           long colRangeEnd,
                                           long outputID)

Copy federated mapping while giving the federated data new IDs and setting the ranges from zero to row and column ends specified. The overlapping ranges are given an overlap number to separate the ranges when putting to the federated map. The federation map returned is of type FType.PART.

Parameters:

rowRangeEnd - end of range for the rows

colRangeEnd - end of range for the columns

outputID - ID given to the output

Returns:

new federation map with overlapping ranges with partially aggregated values

bind

public FederationMap bind(long rOffset,
                          long cOffset,
                          FederationMap that)

transpose

public FederationMap transpose()

getMaxIndexInRange

public long getMaxIndexInRange(int dim)

forEachParallel

public void forEachParallel(BiFunction<FederatedRange,FederatedData,Void> forEachFunction)

Execute a function for each FederatedRange + FederatedData pair. The function should not change any data of the pair and instead use mapParallel if that is a necessity. Note that this operation is parallel and necessary synchronisation has to be performed.

Parameters:

forEachFunction - function to execute for each pair

mapParallel

public FederationMap mapParallel(long newVarID,
                                 BiFunction<FederatedRange,FederatedData,Void> mappingFunction)

Execute a function for each FederatedRange + FederatedData pair mapping the pairs to their new form by directly changing both FederatedRange and FederatedData. The varIDs don't have to be changed by the mappingFunction as that will be done by this method. Note that this operation is parallel and necessary synchronisation has to be performed.

Parameters:

newVarID - the new varID to be used by the new FederationMap

mappingFunction - the function directly changing ranges and data

Returns:

the new FederationMap

filter

public FederationMap filter(IndexRange ixrange)

reverseFedMap

public void reverseFedMap()

toString

public String toString()

Overrides:

toString in class Object

clone

public FederationMap clone()

Overrides:

clone in class Object