EvaluationManager

class EvaluationManager(metric_config: MetricManager)

Coordinator for evaluation operations.

Manages services and evaluators for model evaluation and visualization. Services implement functionality shared by all evaluators. Evaluators implement a specific evaluation method. EvaluationManager coordinates the use of services and evaluators.

Parameters:

metric_configMetricManager

The metric configuration manager for handling evaluation metrics

Attributes:

servicesServiceBundle

The global services bundle providing shared functionality

metric_configMetricManager

The metric configuration manager

output_dirPath or None

The output directory for the evaluation results

registryEvaluatorRegistry

The evaluator registry with evaluators for models

Notes

The EvaluationManager serves as the central coordinator for all evaluation operations. It initializes built-in and custom evaluators, provides access to the services bundle, and provides methods for model saving/loading and evaluator retrieval.

Examples

Initialize evaluation manager:

>>> from brisk.evaluation import metric_manager
>>> metric_mgr = metric_manager.MetricManager()
>>> eval_mgr = EvaluationManager(metric_mgr)

Get an evaluator:

>>> evaluator = eval_mgr.get_evaluator("classification_metrics")

get_evaluator(name: str) → BaseEvaluator

Return an evaluator instance.

Retrieves an evaluator from the registry and configures it with the current metric configuration.

Parameters:

namestr

The name of the evaluator to retrieve

Returns:

BaseEvaluator

An evaluator instance configured with the current metric settings

Notes

The returned evaluator is configured with the current metric configuration, making it ready for use in evaluation operations.

load_model(filepath: str) → BaseEstimator

Load model from pickle file.

Loads a previously saved model from a pickle file. The loaded model package includes both the model and its metadata.

Parameters:

filepathstr

Path to the saved model file

Returns:

BaseEstimator

The loaded model object

Raises:

FileNotFoundError

If the model file does not exist at the specified path

Notes

This method loads the complete model package that was saved using the save_model method, which includes both the model and metadata.

save_model(model: BaseEstimator, filename: str) → None

Save model to pickle file.

Saves a trained model along with its metadata to a pickle file in the current output directory.

Parameters:

modelBaseEstimator

The trained model to save

filenamestr

The name for the output file (without extension)

Returns:

None

Notes

The saved model package includes both the model object and its metadata. The output directory is created if it doesn’t exist. The file is saved with a .pkl extension.

set_experiment_values(output_dir: str, split_metadata: Dict[str, Any], group_index_train: Dict[str, array], group_index_test: Dict[str, array]) → None

Update services and metric_config with the values for the current experiment.

Sets up the evaluation environment for a specific experiment by configuring the output directory, updating experiment configuration, and setting split metadata for metrics.

Parameters:

output_dirstr

The output directory for the evaluation results

split_metadataDict[str, Any]

The split metadata for the current experiment containing information about data splits

group_index_trainDict[str, np.array]

The group index for the training split

group_index_testDict[str, np.array]

The group index for the testing split

Returns:

None

Notes

This method must be called before running evaluations to ensure that all services and metrics are properly configured for the current experiment context.

update_metrics(split_metadata: Dict[str, Any]) → None

Update the metric configuration with the split metadata.

Configures the metric manager with information about the current data splits to ensure proper metric calculation.

Parameters:

split_metadataDict[str, Any]

The split metadata for the current experiment containing information about data splits

Returns:

None

Notes

This method updates the internal metric configuration to reflect the current experiment’s data split characteristics.