MetricManager

class MetricManager(*metric_wrappers)

A class to manage scoring metrics.

Provides access to various scoring metrics, allowing retrieval by either their full names or common abbreviations. Supports both built-in and custom metrics with comprehensive name resolution and configuration export.

Parameters:

*metric_wrappersMetricWrapper

Instances of MetricWrapper for each metric to include

Attributes:

_metrics_by_namedict

Dictionary mapping metric names to MetricWrapper instances

_abbreviations_to_namedict

Dictionary mapping metric abbreviations to full names

_display_name_to_namedict

Dictionary mapping metric display names to full names

Notes

The MetricManager provides a unified interface for accessing metrics regardless of whether they are built-in or custom. It maintains mappings for name resolution and supports both function and scorer retrieval patterns.

Examples

Create a metric manager with built-in metrics:

>>> from brisk.defaults import regression_metrics
>>> manager = MetricManager(*regression_metrics.REGRESSION_METRICS)

Get a metric function:

>>> metric_func = manager.get_metric("mean_squared_error")

Get a scorer callable:

>>> scorer = manager.get_scorer("mse")  # Using abbreviation

export_params() → List[Dict[str, Any]]

Export metric configuration data for rerun functionality.

Detects built-in metric collections and exports custom metrics with their function definitions to enable exact rerun functionality.

Returns:

List[Dict[str, Any]]

List of metric configurations that can be used to recreate the MetricManager

Notes

For built-in collections, exports collection references. For custom metrics, it exports the full configuration including function definitions.

The exported configuration can be used to recreate the exact same MetricManager instance, enabling reproducible experiments.

get_metric(identifier: str) → Callable

Retrieve a metric function by name or abbreviation.

Gets the metric function with its parameters configured for the current context.

Parameters:

identifierstr

Full name, abbreviation, or display name of the metric

Returns:

Callable

The metric function with parameters configured

Raises:

ValueError

If metric identifier is not found

Notes

The returned function includes any parameters that were set via set_split_metadata or other configuration methods.

get_name(identifier: str) → str

Retrieve a metric’s display name.

Gets the formatted display name for the metric, which is typically used for reporting and visualization.

Parameters:

identifierstr

Full name, abbreviation, or display name of the metric

Returns:

str

The formatted display name

Raises:

ValueError

If metric identifier is not found

Notes

The display name is typically a human-readable version of the metric name suitable for reports and plots.

get_scorer(identifier: str) → Callable

Retrieve a scoring callable by name or abbreviation.

Gets the scorer callable that can be used with scikit-learn’s cross-validation and other evaluation functions.

Parameters:

identifierstr

Full name, abbreviation, or display name of the metric

Returns:

Callable

The scoring callable

Raises:

ValueError

If metric identifier is not found

Notes

The scorer is suitable for use with scikit-learn’s make_scorer and cross-validation functions.

is_higher_better(identifier: str) → bool

Determine if a higher value is better for this metric.

Checks whether higher values of the metric indicate better performance, which is important for optimization and comparison.

Parameters:

identifierstr

Full name, abbreviation, or display name of the metric

Returns:

bool

True if a higher value is better for this metric, False otherwise

Raises:

ValueError

If metric identifier is not found

Notes

This information is crucial for optimization algorithms and model selection, as it determines the direction of optimization.

list_metrics() → List[str]

Get list of available metric names.

Returns all metric names that are currently registered in the manager.

Returns:

List[str]

List of available metric names

Notes

This returns the full names of all registered metrics, not abbreviations or display names.

set_split_metadata(split_metadata: Dict[str, Any]) → None

Set the split_metadata for all metrics.

Updates all registered metrics with the provided split metadata, which may be used by metrics that need information about the data splits.

Parameters:

split_metadataDict[str, Any]

Metadata to set for all metrics

Returns:

None

Notes

This method propagates the split metadata to all registered metric wrappers, allowing them to use this information in their calculations.