Two features are correlated at 0.98 and jointly carry the only signal in the data. Explain what happens to their impurity-based importance and their permutation importance in a random forest, and give a measure that does not mislead here.

Two features are correlated at 0.98 and jointly carry the only signal in the data. Explain what happens to their impurity-based importance and their permutation importance in a random forest, and give a measure that does not mislead here.

Approach: Ask what each measure holds fixed. Split-based importance divides credit between substitutable features, while permutation changes one column while its near-copy stays informative.

Impurity importance splits the credit between the two features so each looks half as important as the signal is, permutation importance drives both toward zero because shuffling one leaves its near-copy able to reconstruct the prediction, and a grouped permutation that shuffles the correlated pair together is the measure that reports the truth. Impurity importance sums the criterion reduction over splits on each feature, and because the trees choose between two interchangeable columns roughly at random, the real importance is divided and neither ranks where it should. Permutation importance measures the loss increase when one column is shuffled, which is a statement about that column given the others are intact, so a redundant feature scores near zero however predictive it is. That is the failure that produces a research note recommending a feature be dropped because its importance is small. Permutation importance computed on training data measures memorisation rather than signal, so it must be run out of sample. The fix is to cluster features by correlation and permute each cluster jointly, reporting importance per cluster. Conditional permutation and Shapley values with a background distribution address the same problem at higher cost.

Follow-up: With ten features in one correlated cluster and three independent features, how does grouped permutation importance change the ranking, and how do you choose the clustering threshold?

Key concepts: permutation importance, impurity importance, correlated features, grouped importance.