Common Plots#

class PlotLearningCurve(method_name: str, description: str, plot_settings)[source]#

Bases: PlotEvaluator

Plot learning curves showing model performance vs training size.

This evaluator creates learning curve plots that show how model performance changes as the training set size increases. Learning curves are useful for diagnosing bias vs variance problems and determining if more training data would improve performance.

Parameters:

method_namestr: The name of the evaluator (inherited from PlotEvaluator)
descriptionstr: The description of the evaluator output (inherited from PlotEvaluator)
plot_settingsPlotSettings: The plot settings containing theme and color configuration

Attributes:

method_namestr: The name of the evaluator
descriptionstr: The description of the evaluator output
themeAny: The plot theme for styling plots (inherited from PlotEvaluator)
primary_colorstr: Primary color for plots (inherited from PlotEvaluator)
secondary_colorstr: Secondary color for plots (inherited from PlotEvaluator)
accent_colorstr: Accent color for plots (inherited from PlotEvaluator)

Notes

Learning curves show both training and validation scores across different training set sizes. The gap between training and validation scores can indicate overfitting (large gap) or underfitting (both scores are low).

The evaluator uses cross-validation to get robust estimates of performance at each training size, with error bars showing the standard deviation across folds.

Examples

Use the learning curve evaluator:

>>> from brisk.evaluation.evaluators import registry
>>> evaluator = registry.get("brisk_plot_learning_curve")
>>> evaluator.plot(model, X, y, "learning_curve", cv=5)

generate_plot_data(model: BaseEstimator, X: DataFrame, y: Series, cv: int = 5, num_repeats: int | None = None, n_jobs: int = -1, metric: str = 'neg_mean_absolute_error') → Dict[str, Any][source]#

Calculate the plot data for the learning curve.

Generates learning curve data using cross-validation across different training set sizes.

Parameters:

modelbase.BaseEstimator: Model to evaluate
Xpd.DataFrame: Training features
ypd.Series: Training target values
cvint, optional: Number of cross-validation folds, by default 5
num_repeatsint, optional: Number of times to repeat CV, by default None
n_jobsint, optional: Number of parallel jobs, by default -1
metricstr, optional: Scoring metric to use, by default “neg_mean_absolute_error”

Returns:

Dict[str, Any]: A dictionary containing the learning curve data with keys: - train_sizes: Array of training set sizes - train_scores_mean: Mean training scores across folds - train_scores_std: Standard deviation of training scores - test_scores_mean: Mean validation scores across folds - test_scores_std: Standard deviation of validation scores - fit_times_mean: Mean fit times across folds - fit_times_std: Standard deviation of fit times

Notes

The method uses the utility service to get the appropriate cross-validation splitter based on the data characteristics. The learning curve is calculated with 5 training size points ranging from 10% to 100% of the dataset.

plot(model: BaseEstimator, X: DataFrame, y: Series, filename: str = 'learning_curve', cv: int = 5, num_repeats: int = 1, n_jobs: int = -1, metric: str = 'neg_mean_absolute_error') → None[source]#

Plot learning curves showing model performance vs training size.

Executes the complete learning curve plotting workflow. This includes calculating learning curve data using cross-validation, creating the plot, and saving the results.

Parameters:

modelbase.BaseEstimator: Model to evaluate
Xpd.DataFrame: Training features
ypd.Series: Training target values
filenamestr, optional: Name for output file, by default “learning_curve”
cvint, optional: Number of cross-validation folds, by default 5
num_repeatsint, optional: Number of times to repeat CV, by default 1
n_jobsint, optional: Number of parallel jobs, by default -1
metricstr, optional: Scoring metric to use, by default “neg_mean_absolute_error”

Returns:

None

Notes

The learning curve is calculated using scikit-learn’s learning_curve function with the specified cross-validation strategy. The plot shows both training and validation scores with error bars representing the standard deviation across folds.

The training sizes are automatically determined as fractions of the total dataset size, ranging from 10% to 100%.

class PlotFeatureImportance(method_name: str, description: str, plot_settings)[source]#

Bases: PlotEvaluator

Plot the feature importance for the model.

This evaluator creates feature importance plots using either built-in feature importance (for tree-based models) or permutation importance (for any model). Feature importance helps identify which features contribute most to the model’s predictions.

Parameters:

method_namestr: The name of the evaluator (inherited from PlotEvaluator)
descriptionstr: The description of the evaluator output (inherited from PlotEvaluator)
plot_settingsPlotSettings: The plot settings containing theme and color configuration

Attributes:

method_namestr: The name of the evaluator
descriptionstr: The description of the evaluator output
themeAny: The plot theme for styling plots (inherited from PlotEvaluator)
primary_colorstr: Primary color for plots (inherited from PlotEvaluator)
secondary_colorstr: Secondary color for plots (inherited from PlotEvaluator)
accent_colorstr: Accent color for plots (inherited from PlotEvaluator)

Notes

For tree-based models (DecisionTree, RandomForest, GradientBoosting), the evaluator uses the built-in feature_importances_ attribute. For other models, it uses permutation importance, which measures the decrease in model performance when a feature is randomly shuffled.

The threshold parameter can be either an integer (number of top features) or a float (proportion of features to keep).

Examples

Use the feature importance evaluator:

>>> from brisk.evaluation.evaluators import registry
>>> evaluator = registry.get("brisk_plot_feature_importance")
>>> evaluator.plot(
...     model, X, y, threshold=10, feature_names=feature_names, 
...     filename="importance", metric="accuracy", num_rep=5
    )

generate_plot_data(model: BaseEstimator, X: DataFrame, y: Series, threshold: int | float, feature_names: List[str], metric: str, num_rep: int) → Tuple[DataFrame, float, float][source]#

Calculate the plot data for the feature importance.

Calculates feature importance using either built-in importance or permutation importance, then filters features based on the threshold parameter.

Parameters:

modelbase.BaseEstimator: The model to evaluate
Xpd.DataFrame: The input features
ypd.Series: The target data
thresholdUnion[int, float]: The number of features or the threshold to filter features by importance
feature_namesList[str]: A list of feature names corresponding to the columns in X
metricstr: The metric to use for evaluation
num_repint: The number of repetitions for calculating importance

Returns:

Tuple[pd.DataFrame, float, float]: A tuple containing: - Feature importance DataFrame with Feature and Importance columns - Plot width (float) - Plot height (float)

Notes

For tree-based models, the method uses the built-in feature_importances_ attribute. For other models, it uses scikit-learn’s permutation_importance function with the specified number of repetitions.

The threshold filtering is applied after importance calculation: - If threshold is int: keeps top N features - If threshold is float: keeps top proportion of features

plot(model: BaseEstimator, X: DataFrame, y: Series, threshold: int | float, feature_names: List[str], filename: str, metric: str, num_rep: int) → None[source]#

Plot the feature importance for the model and save the plot.

Executes the complete feature importance plotting workflow. This includes calculating feature importance, filtering features, creating the plot, and saving the results.

Parameters:

modelbase.BaseEstimator: The model to evaluate
Xpd.DataFrame: The input features
ypd.Series: The target data
thresholdUnion[int, float]: The number of features or the threshold to filter features by importance. If int, shows top N features. If float, shows top proportion of features.
feature_namesList[str]: A list of feature names corresponding to the columns in X
filenamestr: The name of the output file (without extension)
metricstr: The metric to use for evaluation (used for permutation importance)
num_repint: The number of repetitions for calculating importance

Returns:

None

Notes

The method automatically chooses between built-in feature importance (for tree-based models) and permutation importance (for other models) based on the model type.

The plot dimensions are automatically adjusted based on the number of features to ensure readability.

class PlotModelComparison(method_name: str, description: str, plot_settings)[source]#

Bases: PlotEvaluator

Plot a comparison of multiple models based on the specified measure.

This evaluator creates bar charts comparing the performance of multiple models on a single metric. This is useful for model selection and performance comparison across different algorithms.

Parameters:

method_namestr: The name of the evaluator (inherited from PlotEvaluator)
descriptionstr: The description of the evaluator output (inherited from PlotEvaluator)
plot_settingsPlotSettings: The plot settings containing theme and color configuration

Attributes:

method_namestr: The name of the evaluator
descriptionstr: The description of the evaluator output
themeAny: The plot theme for styling plots (inherited from PlotEvaluator)
primary_colorstr: Primary color for plots (inherited from PlotEvaluator)
secondary_colorstr: Secondary color for plots (inherited from PlotEvaluator)
accent_colorstr: Accent color for plots (inherited from PlotEvaluator)

Notes

This evaluator is particularly useful for model selection and performance comparison. It evaluates all models on the same dataset using the same metric, ensuring fair comparison.

The plot shows model names on the x-axis and scores on the y-axis, with score values displayed on top of each bar for easy reading.

Examples

Compare multiple models:

>>> from brisk.evaluation.evaluators import registry
>>> evaluator = registry.get("brisk_plot_model_comparison")
>>> evaluator.plot(model1, model2, model3, X=X, y=y, 
...                metric="accuracy", filename="comparison")

generate_plot_data(*models: BaseEstimator, X: DataFrame, y: Series, metric: str) → DataFrame[source]#

Calculate the plot data for the model comparison.

Evaluates each model on the specified metric and prepares data for the comparison plot.

Parameters:

*modelsbase.BaseEstimator: A variable number of model instances to evaluate
Xpd.DataFrame: The input features for evaluation
ypd.Series: The target data for evaluation
metricstr: The metric to evaluate and plot

Returns:

pd.DataFrame or None: A dataframe containing the model names and their scores, or None if evaluation failed

Notes

The method gets the display names for each model from their algorithm wrappers and evaluates each model using the specified metric. Scores are rounded to 3 decimal places for display.

If any model fails to evaluate (e.g., metric not found), the method returns None and logs a warning.

plot(*models: BaseEstimator, X: DataFrame, y: Series, metric: str, filename: str) → None[source]#

Plot a comparison of multiple models based on the specified metric.

Executes the complete model comparison plotting workflow. This includes evaluating each model on the specified metric, creating the comparison plot, and saving the results.

Parameters:

*modelsbase.BaseEstimator: A variable number of model instances to evaluate
Xpd.DataFrame: The input features for evaluation
ypd.Series: The target data for evaluation
metricstr: The metric to evaluate and plot
filenamestr: The name of the output file (without extension)

Returns:

None

Notes

The method evaluates each model individually on the same dataset using the same metric, ensuring fair comparison. If any model fails to evaluate, the plotting process is aborted.

Results are saved with metadata for later analysis and reporting.

class PlotShapleyValues(method_name: str, description: str, plot_settings)[source]#

Bases: PlotEvaluator

Plot SHAP (SHapley Additive exPlanations) values for feature importance.

This evaluator creates SHAP value plots for model interpretability. SHAP values provide a unified framework for explaining model predictions by quantifying the contribution of each feature to individual predictions.

Parameters:

method_namestr: The name of the evaluator (inherited from PlotEvaluator)
descriptionstr: The description of the evaluator output (inherited from PlotEvaluator)
plot_settingsPlotSettings: The plot settings containing theme and color configuration

Attributes:

method_namestr: The name of the evaluator
descriptionstr: The description of the evaluator output
themeAny: The plot theme for styling plots (inherited from PlotEvaluator)
primary_colorstr: Primary color for plots (inherited from PlotEvaluator)
secondary_colorstr: Secondary color for plots (inherited from PlotEvaluator)
accent_colorstr: Accent color for plots (inherited from PlotEvaluator)

Notes

SHAP values provide model-agnostic explanations that are consistent and locally accurate. The evaluator supports multiple plot types: - bar: Mean absolute SHAP values across all samples - waterfall: Feature contributions for a single sample - violin: Distribution of SHAP values across samples - beeswarm: Scatter plot of SHAP values with jitter

The evaluator automatically chooses the appropriate SHAP explainer based on the model type (TreeExplainer, LinearExplainer, or KernelExplainer).

Examples

Use the SHAP values evaluator:

>>> from brisk.evaluation.evaluators import registry
>>> evaluator = registry.get("brisk_plot_shapley_values")
>>> evaluator.plot(model, X, y, "shap_values", plot_type="bar")

generate_plot_data(model: BaseEstimator, X: DataFrame, y: Series) → Dict[str, Any] | None[source]#

Generate SHAP values and prepare data for plotting.

Calculates SHAP values using the appropriate explainer for the model type and prepares data for plotting.

Parameters:

modelbase.BaseEstimator: Trained model to explain
Xpd.DataFrame: Feature data
ypd.Series: Target data

Returns:

Dict[str, Any] or None: Dictionary containing SHAP values and related data, or None if failed

Notes

The method automatically chooses the appropriate SHAP explainer: - TreeExplainer: For tree-based models (RandomForest, XGBoost, etc.) - LinearExplainer: For linear models (LogisticRegression, LinearRegression) - KernelExplainer: For any model (model-agnostic but slower)

If SHAP is not installed, the method returns None and logs a warning.

plot(model: BaseEstimator, X: DataFrame, y: Series, filename: str = 'shap_values', plot_type: str = 'bar') → None[source]#

Generate SHAP value plots for feature importance.

Executes the complete SHAP plotting workflow. This includes generating SHAP values, creating the specified plot type(s), and saving the results.

Parameters:

modelbase.BaseEstimator: Trained model to explain
Xpd.DataFrame: Feature data for generating explanations
ypd.Series: Target data (not used for SHAP but required by interface)
filenamestr, optional: Base output filename, by default “shap_values”
plot_typestr, optional: Type of SHAP plot (“bar”, “waterfall”, “violin”, “beeswarm”). Multiple types can be specified as “bar,waterfall” to generate multiple plots, by default “bar”

Returns:

None

Notes

The method supports multiple plot types in a single call by specifying comma-separated plot types (e.g., “bar,waterfall”). Each plot type is saved as a separate file.

If SHAP is not installed, the method will log a warning and skip SHAP plot generation.

Common Plots#

This Page