Model Evaluators#

class MeasureEvaluator(method_name: str, description: str)[source]#

Bases: BaseEvaluator

Template for evaluators that calculate measures of model performance.

Abstract base class for evaluators that calculate performance measures and metrics for machine learning models. Provides a standardized workflow for model evaluation including prediction generation, metric calculation, metadata handling, and result saving.

Parameters:

method_namestr: The name of the evaluator
descriptionstr: The description of the evaluator output

Attributes:

method_namestr: The name of the evaluator
descriptionstr: The description of the evaluator output
servicesServiceBundle or None: The global services bundle
metric_configMetricManager or None: The metric configuration manager
primary_colorstr: Primary color for plots and visualizations
secondary_colorstr: Secondary color for plots and visualizations
accent_colorstr: Accent color for plots and visualizations

Notes

This abstract base class provides a template for implementing model performance evaluation methods. Subclasses must implement the _calculate_measures method to define the specific metric calculation logic.

The class handles the complete evaluation workflow: 1. Generate predictions using the model 2. Calculate measures using _calculate_measures method 3. Generate metadata for the results 4. Save results to JSON file with metadata 5. Log the results

Examples

Create a custom measure evaluator:

>>> class CustomMeasureEvaluator(MeasureEvaluator):
...     def __init__(self):
...         super().__init__("custom_measure", "a custom measure")
...     
...     def _calculate_measures(self, predictions, y_true, metrics):
...         # Custom measure calculation logic
...         return {"custom_metric": 0.85}

abstractmethod calculate_measures(predictions: Series, y_true: Series, metrics: List[str]) → Dict[str, float][source]#

Must implement this method to calculate something.

Abstract method that must be implemented by subclasses to define the specific measure calculation logic. This is where the actual metric computation takes place.

Parameters:

predictionspd.Series: The model predictions
y_truepd.Series: The true target values
metricsList[str]: The list of metric names to calculate

Returns:

Dict[str, float]: Dictionary containing the calculated measures with metric names as keys and their values as values

Notes

This method must be implemented by all subclasses. It should contain the specific logic for calculating the requested metrics using the provided predictions and true values.

evaluate(model: BaseEstimator, X: DataFrame, y: Series, metrics: List[str], filename: str) → None[source]#

Template for all measure methods to follow.

Executes the complete evaluation workflow for model performance measures. This method orchestrates the evaluation process by generating predictions, calculating measures, and handling result processing.

Parameters:

modelbase.BaseEstimator: The trained model to evaluate
Xpd.DataFrame: The input data for evaluation
ypd.Series: The true target values
metricsList[str]: The list of metric names to calculate
filenamestr: The name of the file to save the results to (without extension)

Returns:

None

Notes

This method provides the standard workflow for model evaluation: 1. Generate predictions using the model 2. Calculate measures using _calculate_measures 3. Generate metadata for the results 4. Save results to JSON file 5. Log the results

The method delegates the actual measure calculation to the _calculate_measures method, which must be implemented by subclasses.

log_results(results: Dict[str, float], filename: str) → None[source]#

Default logging - can be overridden.

Logs the evaluation results in a standardized format. This method provides default logging functionality that can be overridden by subclasses for custom logging behavior.

Parameters:

resultsDict[str, float]: The results of the evaluation to log
filenamestr: The name of the file where results were saved

Returns:

None

Notes

The default implementation logs all key-value pairs from the results dictionary in a formatted manner. Subclasses can override this method to provide custom logging behavior.

The logging format shows each metric name and its value with 4 decimal places precision.

report(results: Dict[str, Any]) → Tuple[List[str], List[List[Any]]][source]#

Default reporting - can be overridden.

Converts evaluation results into a format suitable for reporting. By default, assumes that the keys of the results dictionary are column headers and values are lists corresponding to each row.

Parameters:

resultsDict[str, Any]: The results of the evaluation

Returns:

Tuple[List[str], List[List[Any]]]: A tuple containing: - List of column headers - Nested list of rows (each row is a list of values)

Notes

The default implementation assumes a specific results format where keys are column headers and values are lists of row data. If the results are in a different format, this method should be overridden to return the appropriate column headers and row data.

The metadata key is automatically excluded from the column headers.

class PlotEvaluator(method_name: str, description: str, plot_settings)[source]#

Bases: BaseEvaluator

Template for model evaluators that plot data.

Abstract base class for evaluators that create plots and visualizations from model predictions and data. Provides a standardized workflow for model plotting including plot data generation, plot creation, metadata handling, and result saving.

Parameters:

method_namestr: The name of the evaluation method
descriptionstr: The description of the evaluation method
plot_settingsPlotSettings: The plot settings containing theme and color configuration

Attributes:

method_namestr: The name of the evaluator
descriptionstr: The description of the evaluator output
themeAny: The plot theme for styling plots
primary_colorstr: Primary color for plots (from plot settings)
secondary_colorstr: Secondary color for plots (from plot settings)
accent_colorstr: Accent color for plots (from plot settings)
servicesServiceBundle or None: The global services bundle (inherited from BaseEvaluator)
metric_configMetricManager or None: The metric configuration manager (inherited from BaseEvaluator)

Notes

This abstract base class provides a template for implementing model plotting methods. Subclasses must implement the _generate_plot_data and _create_plot methods to define the specific plotting logic.

The class handles the complete plotting workflow: 1. Generate plot data using _generate_plot_data method 2. Create the plot using _create_plot method 3. Generate metadata for the plot 4. Save the plot with metadata 5. Log the results

The constructor automatically configures matplotlib to use a non-interactive backend for thread safety and applies the provided plot settings.

Examples

Create a custom plot evaluator:

>>> class CustomPlotEvaluator(PlotEvaluator):
...     def __init__(self, plot_settings):
...         super().__init__("custom", "Custom plot", plot_settings)
...     
...     def _generate_plot_data(self, model, X, y, **kwargs):
...         # Custom plot data generation logic
...         return plot_data
...     
...     def _create_plot(self, plot_data, **kwargs):
...         # Custom plot creation logic
...         return plot

abstractmethod generate_plot_data(model: BaseEstimator, X: DataFrame, y: Series, **kwargs) → Any[source]#

MUST implement: Generate data for plotting.

Abstract method that must be implemented by subclasses to define the specific plot data generation logic. This is where the data preparation for plotting takes place.

Parameters:

modelbase.BaseEstimator: The trained model to evaluate
Xpd.DataFrame: The input data for plotting
ypd.Series: The true target values
**kwargs: Additional keyword arguments for plot data generation

Returns:

Any: The data structure to be used by _create_plot method

Notes

This method must be implemented by all subclasses. It should contain the specific logic for preparing data for plotting, such as data transformation, aggregation, or filtering.

The returned data structure should be compatible with the _create_plot method implementation.

log_results(plot_name: str, filename: str) → None[source]#

Default logging - can be overridden.

Logs the plot creation results in a standardized format. This method provides default logging functionality that can be overridden by subclasses for custom logging behavior.

Parameters:

plot_namestr: The name of the plot that was created
filenamestr: The name of the file where the plot was saved

Returns:

None

Notes

The default implementation logs the plot name and file path. Subclasses can override this method to provide custom logging behavior.

The logging includes the full output path with .svg extension.

plot(model: BaseEstimator, X: DataFrame, y: Series, filename: str) → None[source]#

Template for all plot methods to follow.

Executes the complete plotting workflow for model plots. This method orchestrates the plotting process by calling the abstract methods and handling plot processing.

Parameters:

modelbase.BaseEstimator: The trained model to evaluate
Xpd.DataFrame: The input data for plotting
ypd.Series: The true target values
filenamestr: The name of the file to save the plot to (without extension)

Returns:

None

Notes

This method provides the standard workflow for model plotting: 1. Generate plot data using _generate_plot_data 2. Create the plot using _create_plot 3. Generate metadata for the plot 4. Save the plot with metadata 5. Log the results

The method delegates the actual plot data generation and plot creation to the abstract methods, which must be implemented by subclasses.

Model Evaluators#

This Page