EvaluationManager

class EvaluationManager(metric_manager: MetricManager)

Coordinator for evaluation operations.

Manages services and evaluators for model evaluation and visualization. Services implement functionality shared by all evaluators. Evaluators implement a specific evaluation method. EvaluationManager coordinates the use of services and evaluators.

Parameters:

metric_managerMetricManager

The metric configuration manager for handling evaluation metrics

Attributes:

servicesServiceBundle

The global services bundle providing shared functionality

metric_managerMetricManager

The metric configuration manager

output_dirPath or None

The output directory for the evaluation results

registryEvaluatorRegistry

The evaluator registry with evaluators for models

Notes

The EvaluationManager serves as the central coordinator for all evaluation operations. It initializes built-in and custom evaluators, provides access to the services bundle, and provides methods for model saving/loading and evaluator retrieval.

Examples

Initialize evaluation manager:

>>> from brisk.evaluation import metric_manager
>>> metric_mgr = metric_manager.MetricManager()
>>> eval_mgr = EvaluationManager(metric_mgr)

Get an evaluator:

>>> evaluator = eval_mgr.get_evaluator("classification_metrics")

get_evaluator(name: str) → BaseEvaluator

Return an evaluator instance.

Retrieves an evaluator from the registry and configures it with the current metric configuration.

Parameters:

namestr

The name of the evaluator to retrieve

Returns:

BaseEvaluator

An evaluator instance configured with the current metric settings

Notes

The returned evaluator is configured with the current metric configuration, making it ready for use in evaluation operations.

initialize_evaluators() → None

Initialize all evaluators with shared services.

This method registers all built-in evaluators with the evaluator registry and sets the services for each evaluator. It also attempts to register any custom evaluators from the project.

Returns:

None

Notes

The initialization process includes: 1. Registering built-in evaluators with plot settings 2. Attempting to register custom evaluators 3. Setting services for all evaluators 4. Configuring the reporting service with the evaluator registry

load_model(filepath: str) → BaseEstimator

Load model from pickle file.

Loads a previously saved model from a pickle file. The loaded model package includes both the model and its metadata.

Parameters:

filepathstr

Path to the saved model file

Returns:

BaseEstimator

The loaded model object

Raises:

FileNotFoundError

If the model file does not exist at the specified path

Notes

This method loads the complete model package that was saved using the save_model method, which includes both the model and metadata.

save_model(model: BaseEstimator, filename: str) → None

Save model to pickle file.

Saves a trained model along with its metadata to a pickle file in the current output directory.

Parameters:

modelBaseEstimator

The trained model to save

filenamestr

The name for the output file (without extension)

Returns:

None

Notes

The saved model package includes both the model object and its metadata. The output directory is created if it doesn’t exist. The file is saved with a .pkl extension.

set_evaluator_registry()

set_evaluator_services() → None

set_experiment_config(group_index_train: Dict[str, array], group_index_test: Dict[str, array], update_method: Callable | None = None) → None

Update services and metric_manager with the values for the current experiment.

Parameters:

group_index_trainDict[str, np.array]

The group index for the training split

group_index_testDict[str, np.array]

The group index for the testing split

update_method: Optional[Callable]

Inject a different update function; used for testing

Returns:

None

Notes

This method must be called before running evaluations to ensure that all services and metrics are properly configured for the current experiment context.

set_output_dir(output_dir: str) → None

Parameters:

output_dirstr

The output directory for the evaluation results

Returns:

None

set_services(services: ServiceBundle | None = None) → None

update_metrics(split_metadata: Dict[str, Any]) → None

Update the metric configuration with the split metadata.

Configures the metric manager with information about the current data splits to ensure proper metric calculation.

Parameters:

split_metadataDict[str, Any]

The split metadata for the current experiment containing information about data splits

Returns:

None

Notes

This method updates the internal metric configuration to reflect the current experiment’s data split characteristics.