pyBIA.feature_selection

Classes

BorutaSHAP

Feature selection wrapper combining Boruta and SHAP-based importance metrics.

Module Contents

class pyBIA.feature_selection.BorutaSHAP(model=None, importance_measure='Shap', classification=True, percentile=100, pvalue=0.05)

Feature selection wrapper combining Boruta and SHAP-based importance metrics.

The BorutaSHAP class extends the Boruta feature selection methodology using model-specific importance metrics (SHAP, Gini, or permutation-based), making it compatible with both classification and regression tasks. It introduces a shadow feature mechanism to iteratively compare feature importances against randomized features, statistically testing for their relevance. This class supports flexible importance measures, sample-based selection using Isolation Forest, and automatic integration with scikit-learn-style pipelines via fit, transform, and set_params.

Parameters:

• model (object, optional) – A scikit-learn-compatible model with fit and predict methods. If not provided, a default RandomForestClassifier or RandomForestRegressor is used depending on the task type.

• importance_measure (str, default='Shap') – Metric used to compute feature importances. One of [‘Shap’, ‘gini’, ‘perm’].

• classification (bool, default=True) – Whether the task is classification (True) or regression (False).

• percentile (int, default=100) – Percentile of shadow feature importances to use as a threshold for significance testing. Lower values make the algorithm more lenient.

• pvalue (float, default=0.05) – Significance level for hypothesis testing. Lower values result in stricter feature rejection, potentially increasing runtime.

accepted

Final list of features accepted as important.

Type:

list

rejected

Final list of features rejected as unimportant.

Type:

list

tentative

List of features whose importance is undetermined.

Type:

list

history_x

DataFrame recording feature importances over each iteration.

Type:

pd.DataFrame

history_shadow

Historical shadow feature importances.

Type:

np.ndarray

hits

Array tracking the number of times each feature beats the shadow threshold.

Type:

np.ndarray

all_columns

List of original column names in X.

Type:

np.ndarray

X_boruta

DataFrame containing both original and shadow features.

Type:

pd.DataFrame

fit(X, y, ...)

Runs the BorutaSHAP feature selection algorithm on the provided dataset.

results_to_csv(filename='feature_importance')

Saves the importance scores and feature decisions to CSV.

create_mapping_of_features_to_attribute(maps)

Creates a dictionary mapping each feature to a visual label (e.g., color or decision).

Notes

SHAP-based explanations are intended for tree-based models, such as XGBoost, LightGBM, or CatBoost.

Permutation importance is computed using scikit-learn’s permutation_importance function.

Gini importance requires the model to expose a feature_importances_ attribute, such as in RandomForest models.

Missing values are supported only for models that can handle them internally. Otherwise, a ValueError will be raised.

When sample-based SHAP explanations are enabled, the method uses an Isolation Forest to select a representative subset of the data that preserves the original anomaly score distribution.

References

• Kursa, M. B., & Rudnicki, W. R. (2010). Feature selection with the Boruta package.

• Lundberg, S. M., & Lee, S. I. (2017). A unified approach to interpreting model predictions (SHAP).

• https://github.com/Ekeany/Boruta-Shap

importance_measure = 'Shap'

percentile = 100

pvalue = 0.05

classification = True

model = None

classmethod _get_param_names()

Retrieve the parameter names from the class constructor (__init__).

This utility method introspects the constructor signature to extract all explicitly defined parameter names, excluding ‘self’ and variable keyword arguments (**kwargs). It ensures compatibility with scikit-learn-style estimators by enforcing that no variable positional arguments (*args) are used.

Returns:

A sorted list of parameter names defined in the constructor.

Return type:

list of str

Raises:

RuntimeError – If the class defines variable positional arguments (*args), which violates scikit-learn’s estimator API convention.

get_params(deep=True)

Get the parameters of this estimator.

This method returns a dictionary of all parameters in the estimator. If deep=True, it will recursively retrieve parameters of sub-estimators (i.e., model attributes that implement get_params themselves), using a double-underscore naming convention.

Parameters:

deep (bool, default=True) – If True, include parameters from nested objects (such as the wrapped model). If False, only return parameters directly set on this estimator.

Returns:

params – Dictionary mapping parameter names to their current values. Nested parameters are flattened using the format: ‘component__param’.

Return type:

dict

set_params(**params)

Set the parameters of this estimator.

This method updates the estimator’s parameters using the provided dictionary. It supports both top-level parameters and nested parameters using the scikit-learn convention of double underscores (e.g., ‘model__n_estimators’) for sub-estimators.

Parameters:

**params (dict) – Dictionary of parameter names mapped to their new values. Nested parameters can be updated using double-underscore notation.

Returns:

self – The updated estimator instance.

Return type:

object

Raises:

ValueError – If a parameter name is invalid or does not match any parameter in the estimator.

check_model()

Validate and initialize the model used for feature importance evaluation.

If no model was provided at initialization, this method assigns a default RandomForestClassifier (for classification) or RandomForestRegressor (for regression). It also checks that the model has the required methods and attributes based on the selected importance measure.

Return type:

None

Raises:

AttributeError – If the provided model does not implement both fit and predict, or if the gini importance measure is selected but the model lacks the feature_importances_ attribute.

check_X()

Verify that the input feature data X is a pandas DataFrame.

This method ensures that the feature matrix provided to the BorutaSHAP instance is of the correct type before proceeding with feature selection.

Return type:

None

Raises:

AttributeError – If X is not a pandas DataFrame.

missing_values_y()

Check for missing values in the target variable y.

Supports pandas Series, DataFrame, or NumPy array inputs. Returns True if any missing values are found.

Returns:

True if y contains missing values, False otherwise.

Return type:

bool

Raises:

AttributeError – If y is not a pandas Series, DataFrame, or NumPy array.

check_missing_values()

Check for missing values in the feature matrix X and target variable y.

This method verifies that no missing values are present in the input data. If missing values are found, a warning is issued for models that support them (e.g., XGBoost, LightGBM, CatBoost). Otherwise, a ValueError is raised.

Return type:

None

Raises:

ValueError – If missing values are detected and the model does not support them.

Notes

Models known to support missing values include: XGBoost, CatBoost, and LightGBM.

Check_if_chose_train_or_test_and_train_model()

Split the data and train the model based on the train_or_test strategy.

If train_or_test=’test’, the method splits the Boruta-augmented dataset into training and testing sets (70/30 split) using the specified random_state and optional stratification. The model is trained on the training portion.

If train_or_test=’train’, the model is trained on the full dataset without splitting.

Return type:

None

Raises:

ValueError – If stratification is requested for a regression task, or if train_or_test is not one of the accepted values (“train” or “test”).

Notes

For a detailed discussion on training vs. testing data when computing feature importance, see: https://slds-lmu.github.io/iml_methods_limitations/pfi-data.html

Train_model(X, y, sample_weight=None)

Fit the model to the provided data.

This method trains the model using the given features X and targets y. It handles special cases for certain models like CatBoost, which require categorical feature specifications. It also gracefully handles models that do not accept the verbose parameter.

Parameters:

• X (pandas.DataFrame) – DataFrame containing the feature matrix.

• y (pandas.Series or numpy.ndarray) – Array or Series containing the target variable.

• sample_weight (pandas.Series or numpy.ndarray, optional) – Sample weights to apply during model training.

Return type:

None

fit(X, y, sample_weight=None, n_trials=20, random_state=0, sample=False, train_or_test='test', normalize=True, verbose=True, stratify=None)

Run the BorutaSHAP feature selection process.

This is the core method that performs iterative feature selection by comparing real features against shadow (randomized) features using the chosen importance measure (SHAP, Gini, or permutation). Features are repeatedly tested against the maximum shadow importance, and classified as accepted, rejected, or tentative.

The algorithm proceeds as follows: 1. Extend the dataset by adding shuffled copies of original features (shadow features). 2. Train the model and compute feature importances. 3. Identify features that outperform the maximum shadow importance threshold. 4. Track hit counts and statistically test features against the null hypothesis. 5. Accept, reject, or defer decision on each feature. 6. Repeat until a decision is made for all features or the trial limit is reached.

Parameters:

• X (pandas.DataFrame) – Feature matrix.

• y (pandas.Series or numpy.ndarray) – Target variable.

• sample_weight (pandas.Series or numpy.ndarray, optional) – Observation-level weights used during model training.

• n_trials (int, default=20) – Maximum number of iterations to run the feature selection process.

• random_state (int, default=0) – Random seed for reproducibility.

• sample (bool, default=False) – If True, a representative sample of the data will be selected using Isolation Forest for SHAP value estimation.

• train_or_test ({'train', 'test'}, default='test') – Specifies whether feature importances should be computed on training data or a held-out test split.

• normalize (bool, default=True) – Whether to normalize feature importances using z-score transformation.

• verbose (bool, default=True) – If True, prints the final list of accepted, rejected, and tentative features.

• stratify (array-like, optional) – Class labels used for stratified splitting during train-test division.

Returns:

self – Returns the fitted BorutaSHAP instance.

Return type:

object

Notes

For a detailed discussion on the implications of computing feature importances on training vs. test data, see: https://compstat-lmu.github.io/iml_methods_limitations/pfi-data.html

calculate_rejected_accepted_tentative(verbose)

Finalize feature decisions: accepted, rejected, or tentative.

This method processes the accumulated hit statistics across all trials to determine which features are: - Accepted: consistently more important than shadow features. - Rejected: consistently less important than shadow features. - Tentative: not confidently accepted or rejected.

Parameters:

verbose (bool) – If True, prints the number and names of accepted, rejected, and tentative features.

Return type:

None

create_importance_history()

Initialize arrays to store historical feature importance scores.

This method sets up internal storage for tracking the shadow importances, original feature importances, and cumulative hit counts across all iterations.

Return type:

None

update_importance_history()

Update historical records of feature importances for the current iteration.

This method appends the current shadow and actual feature importances to their respective history arrays, ensuring they remain aligned with the original column order using a mapping index.

Return type:

None

store_feature_importance()

Finalize and store historical feature importance statistics.

This method reshapes the accumulated feature importance history into a pandas DataFrame and appends summary statistics for the shadow features, including maximum, minimum, mean, and median importance values.

Return type:

None

results_to_csv(filename='feature_importance')

Save feature importance summary statistics and decisions to a CSV file.

This method compiles the average and standard deviation of each feature’s importance across all iterations, appends its final classification (Accepted, Rejected, Tentative, or Shadow), and exports the result to disk.

Parameters:

filename (str, default='feature_importance') – The base name for the output CSV file. The file will be saved as ‘<filename>.csv’ in the current working directory.

Return type:

None

remove_features_if_rejected()

Remove rejected features from the dataset.

This method drops features from self.X that have been marked for removal based on the outcome of statistical tests in the current iteration.

Return type:

None

static flatten_list(array)

Flatten a list of lists into a single list.

Parameters:

array (list of lists) – A nested list to be flattened.

Returns:

A single flattened list containing all elements from the sublists.

Return type:

list

create_mapping_between_cols_and_indices()

Create a mapping from feature names to their column indices.

This mapping preserves the original order of columns in self.X and is used to align importance values across iterations.

Returns:

Dictionary mapping column names to their corresponding integer indices.

Return type:

dict

calculate_hits()

Compute hit counts for each feature based on shadow feature comparison.

A feature is assigned a “hit” if its importance exceeds the specified percentile threshold of the shadow feature importances. Hits are padded and aligned to the full column index order.

Returns:

Array of length ncols containing the updated hit counts for each feature.

Return type:

numpy.ndarray

create_shadow_features()

Generate shadow features by shuffling each original feature column.

Shadow features are created by independently permuting each column of the input data X. These are used as a baseline for comparing feature importances. The resulting shadow features are renamed with a ‘shadow_’ prefix and concatenated with the original data to form the extended dataset used during model training.

This method also identifies categorical columns for models (e.g., CatBoost) that require explicit specification of categorical features.

Return type:

None

static calculate_Zscore(array)

Compute the z-score normalization of a numeric array.

Each element is standardized by subtracting the mean and dividing by the standard deviation.

Parameters:

array (array-like) – Input array of numeric values.

Returns:

Z-score normalized values of the input array.

Return type:

list of float

feature_importance(normalize)

Compute feature importance scores for both original and shadow features.

This method calculates importance values based on the specified importance_measure: - ‘shap’: Uses SHAP values computed via shap.TreeExplainer - ‘perm’: Uses permutation importance via sklearn.inspection.permutation_importance - ‘gini’: Uses built-in feature_importances_ from the model (e.g., RandomForest)

Importance values can optionally be normalized using z-score transformation.

Parameters:

normalize (bool) – If True, importance scores are z-score normalized.

Returns:

• X_feature_import (array-like) – Importance scores for the original features.

• Shadow_feature_import (array-like) – Importance scores for the shadow (randomized) features.

Raises:

ValueError – If importance_measure is not one of {‘shap’, ‘perm’, ‘gini’}.

static isolation_forest(X, sample_weight)

Fit an Isolation Forest to the dataset and compute anomaly scores.

This method trains an Isolation Forest on the input feature matrix X and returns anomaly scores for each sample. Higher scores indicate more typical (less anomalous) samples.

Parameters:

• X (pandas.DataFrame or numpy.ndarray) – Input feature matrix.

• sample_weight (array-like) – Sample weights to apply during model fitting.

Returns:

Anomaly scores for each sample, as returned by IsolationForest.score_samples.

Return type:

numpy.ndarray

static get_5_percent(num)

Compute 5 percent of a given number.

Parameters:

num (int or float) – Input number.

Returns:

Value corresponding to 5% of the input, rounded to the nearest integer.

Return type:

int

get_5_percent_splits(length)

Generate index positions at 5% intervals of a given length.

This method returns an array of indices that split a dataset into successive 5% chunks, based on the total number of samples.

Parameters:

length (int) – Total number of samples in the dataset.

Returns:

Array of index positions at 5% intervals.

Return type:

numpy.ndarray

find_sample()

Select a representative sample of the dataset using KS-test on anomaly scores.

This method iteratively draws random samples of increasing size and compares the distribution of their anomaly scores (from Isolation Forest) to the original distribution using the Kolmogorov-Smirnov (KS) test. It starts at 5% of the dataset and increases in 5% increments until a sample is found with a KS p-value > 0.95, indicating statistical similarity.

Returns:

A representative sample of self.X_boruta with similar anomaly score distribution to the full dataset.

Return type:

pandas.DataFrame

explain()

Compute SHAP values for the model using TreeExplainer.

This method uses the SHAP package to calculate feature importances based on Shapley values. It selects TreeExplainer with path-dependent perturbations for efficiency on tree-based models. If self.sample is True, a representative subset of the data (selected via find_sample) is used; otherwise, the full self.X_boruta dataset is used.

For classification tasks, SHAP values across classes are aggregated to compute a single importance value per feature. For regression, absolute SHAP values are averaged directly.

Returns:

The computed SHAP values are stored internally in self.shap_values.

Return type:

None

Raises:

ValueError – If the model is not compatible with SHAP’s TreeExplainer (though in practice, the method currently assumes tree-based models only).

static binomial_H0_test(array, n, p, alternative)

Perform a binomial test for each element in an array.

This method tests the null hypothesis that the probability of success is p in a Bernoulli trial, using a binomial test. Each element in the input array is treated as the number of observed successes out of n trials.

Parameters:

• array (array-like) – Array of observed success counts (can be float; will be rounded).

• n (int) – Number of trials per test.

• p (float) – Null hypothesis probability of success.

• alternative ({'two-sided', 'greater', 'less'}) – Defines the alternative hypothesis.

Returns:

List of p-values from the binomial tests for each element in the input array.

Return type:

list of float

static find_index_of_true_in_array(array)

Return the indices of elements that are True in a boolean array.

Parameters:

array (array-like of bool) – Boolean array indicating which elements to select.

Returns:

Indices where the array has True values.

Return type:

list of int

static bonferoni_corrections(pvals, alpha=0.05, n_tests=None)

Perform statistical tests to accept or reject features based on hit counts.

This method compares the number of times each feature outperformed the shadow features (“hits”) to the expected distribution under the null hypothesis (p = 0.5), using a binomial test. It applies Bonferroni correction to control for multiple comparisons and classifies features as accepted, rejected, or tentative.

Parameters:

iteration (int) – Current iteration number, used as the number of trials in the binomial test.

Returns:

Updates internal attributes: - features_to_remove: list of features to drop in the next iteration. - accepted_columns: list of newly accepted features. - rejected_columns: list of newly rejected features.

Return type:

None

test_features(iteration)

Perform statistical tests to accept or reject features based on accumulated hit counts.

For each feature, this method performs two binomial hypothesis tests: - A right-tailed test to check if the feature is significantly better than random (acceptance). - A left-tailed test to check if the feature is significantly worse than random (rejection).

The tests use the number of times each feature outperformed the shadow feature (stored in self.hits) over iteration trials. Bonferroni correction is applied to control for multiple comparisons.

Parameters:

iteration (int) – The current iteration count, used as the number of trials in the binomial test.

Returns:

Updates the following internal attributes: - self.accepted_columns: list of accepted feature names for this iteration. - self.rejected_columns: list of rejected feature names for this iteration. - self.features_to_remove: list of features to remove from self.X in the next iteration.

Return type:

None

static create_list(array, color)

Create a list of repeated color labels for visualization or mapping.

Parameters:

• array (array-like) – List of elements (used only to determine length).

• color (str) – The color label or string to repeat.

Returns:

A list of the same color repeated to match the length of array.

Return type:

list of str

create_mapping_of_features_to_attribute(maps=[])

Create a dictionary mapping features to attribute labels (e.g., for color or status tagging).

This method maps each feature—tentative, rejected, accepted, and shadow summary features— to a corresponding label or value provided in the maps list. It is typically used for visualization or export purposes.

Parameters:

maps (list of str) – A list of four strings corresponding to labels for: [0] Tentative features [1] Rejected features [2] Accepted features [3] Shadow features (e.g., Max_Shadow, Min_Shadow, etc.)

Returns:

Dictionary mapping each feature name to its corresponding label.

Return type:

dict

static to_dictionary(list_one, list_two)

Create a dictionary by zipping two lists together.

Parameters:

• list_one (list) – List of keys.

• list_two (list) – List of values.

Returns:

Dictionary mapping each element in list_one to the corresponding element in list_two.

Return type:

dict