Package org.apache.sysds.resource.enumeration

Class Enumerator

    • Field Detail

      • DEFAULT_MAX_EXECUTORS

        public static final int DEFAULT_MAX_EXECUTORS
        A reasonable upper bound for the possible number of executors is required to set limits for the search space and to avoid evaluating cluster configurations that most probably would have too high distribution overhead
        See Also:
        Constant Field Values

      • COST_DELTA_FRACTION

        public static final double COST_DELTA_FRACTION
        Time/Monetary delta for considering optimal solutions as fraction
        See Also:
        Constant Field Values

    • Method Detail

      • setCostsWeightFactor

        public static void setCostsWeightFactor​(double newFactor)

      • setMinTime

        public static void setMinTime​(double maxTime)

      • setMinPrice

        public static void setMinPrice​(double maxPrice)

      • setCpuQuota

        public static void setCpuQuota​(int newQuotaValue)

      • preprocessing

        public abstract void preprocessing()
        Called once to enumerate the search space for VM instances for driver or executor nodes. These instances are being represented as

      • processing

        public void processing()
        Called once after preprocessing to fill the pool with optimal solutions by parsing the enumerated search space. Within its execution the number of potential executor nodes is being estimated (enumerated) dynamically for each parsed executor instance.

      • postprocessing

        public EnumerationUtils.SolutionPoint postprocessing()
        Retrieving the estimated optimal configurations after processing.
        Returns:
        optimal cluster configuration and corresponding costs

      • evaluateSingleNodeExecution

        public abstract boolean evaluateSingleNodeExecution​(long driverMemory,
                                                    int cores)

      • estimateRangeExecutors

        public abstract ArrayList<Integer> estimateRangeExecutors​(int driverCores,
                                                          long executorMemory,
                                                          int executorCores)
        Estimates the minimum and maximum number of executors based on given VM instance characteristics, the enumeration strategy and the user-defined configurations
        Parameters:
        driverCores - CPU cores for the currently evaluated driver node
        executorMemory - memory of currently evaluated executor node
        executorCores - CPU cores of currently evaluated executor node
        Returns:
        - [min, max]

      • updateOptimalSolution

        public void updateOptimalSolution​(double newTimeEstimate,
                                  double newMonetaryEstimate,
                                  EnumerationUtils.ConfigurationPoint newPoint)
        Invokes the estimation of the time and monetary cost based on the compiled program and the given cluster configurations. Following the optimization strategy, the given current optimal solution and the new cost estimation, it decides if the given cluster configuration can be potential optimal solution having lower cost or such a cost that is negligibly higher than the current lowest one.
        Parameters:
        newTimeEstimate - estimated time cost for the given configurations
        newMonetaryEstimate - estimated monetary cost for the given configurations
        newPoint - new cluster configuration for estimation

      • getInstances

        public HashMap<String,​CloudInstance> getInstances()
        Meant to be used for testing purposes
        Returns:
        the available instances for enumeration

      • setDriverSpace

        public void setDriverSpace​(EnumerationUtils.InstanceSearchSpace inputSpace)
        Meant to be used for testing purposes
        Parameters:
        inputSpace - the object representing the driver search space

      • getExecutorSpace

        public EnumerationUtils.InstanceSearchSpace getExecutorSpace()
        Meant to be used for testing purposes
        Returns:
        the object representing the executor search space

      • setExecutorSpace

        public void setExecutorSpace​(EnumerationUtils.InstanceSearchSpace inputSpace)
        Meant to be used for testing purposes
        Parameters:
        inputSpace - the object representing the executor search space

      • getEnumStrategy

        public Enumerator.EnumerationStrategy getEnumStrategy()
        Meant to be used for testing purposes
        Returns:
        applied enumeration strategy

      • getCostsWeightFactor

        public double getCostsWeightFactor()
        Meant to be used for testing purposes
        Returns:
        configured weight factor optimization function 'costs'

      • getMaxTime

        public double getMaxTime()
        Meant to be used for testing purposes
        Returns:
        configured max time for consideration (seconds)

      • getMaxPrice

        public double getMaxPrice()
        Meant to be used for testing purposes
        Returns:
        configured max price for consideration (dollars)

      • getOptimalSolution

        public EnumerationUtils.SolutionPoint getOptimalSolution()
        Meant to be used for testing purposes
        Returns:
        current optimal solution