Where the explainers point

The data

Two features, x and y, each drawn uniformly between −1 and 1: 5,000 points in all (1,500 are shown here). Each point’s class comes from a fixed rule with corners and slanted edges. Class A fills the lower left, a sloping band through the middle and everything right of x = 0.4, except for a wedge cut out along the bottom right.

The model

A random forest of 10 trees learns its own version of that rule. The shading is now the model’s prediction at every location. The lines show the true boundary.

It’s close, but it’s built from rectangles, so the slopes become staircases. We know its training algorithm, but not its functional form, which is exactly when we reach for explainers.

Four points

We ask the explainers about four observations. Each sits on or right next to the model’s boundary, where its staircase makes a simple story least likely to hold.

The usual view

Here’s point 1, at (0, 0.1), predicted class A. Explainers usually report back as a bar chart: how important was each feature? Normalised so the top feature scores 1:

Three methods say y, and the counterfactual says x. Which one is right? From the bars alone there’s no way to tell.

Three shared parts

Most local explainers are built from the same three parts:

1. a perturbation mechanism that makes what-if points near the observation (here, a 21 × 21 grid in a 0.2-wide box);
2. an optimisation objective that uses the model’s predictions on those points;
3. a structured output: the explanation.

They differ in what they choose for each part, and that’s where the disagreement comes from. So instead of comparing their numbers, let’s draw what each one found.

SHAP: forces along the axes

SHAP splits a prediction into a contribution from each feature: how far it moves the predicted probability of class A away from the average over the neighbourhood.

We draw each contribution as an arrow parallel to its axis, sized by its magnitude and coloured by the class it pushes towards. At point 3, the whole push comes from x. Its y arrow has zero length.

SHAP’s arrows describe forces on the prediction. They don’t point to where the boundary is.

LIME: a straight line through a bend

LIME fits a simple model (here a LASSO logistic regression, with kumquat) to the model’s predictions on the perturbed grid.

The squares show the random forest’s predictions and the dots show the local model’s. LIME’s explanation is that straight boundary. The arrow at right angles to it shows which feature the local model leans on: mostly y, a little x.

Counterfactuals: the nearest other side

A counterfactual answers “what’s the smallest change that flips the prediction?” We used the What-If approach: the nearest observed point that the model puts in the other class with probability of at least 0.9 (nearest by Gower distance).

Its explanation is a direction. At point 2 the path drops almost straight down (0.05 in y, 0.002 in x), so the counterfactual says y, while the other three methods say x.

Anchors: a box that holds the prediction

Anchors (from kultarr) looks for an if-then rule: a box around the point inside which the model almost always gives the same answer. The box needs a precision of at least 0.95. Ours range from 0.99 to 1.00, and each covers 25% to 43% of its neighbourhood (the dashed square).

A side that has to stay short marks the more important feature, because the box can’t stretch far that way before the prediction changes.

Why they disagree

Back to point 1, with all four drawn together.

Point 1 sits on an edge that slopes up at 0.8, so moving in y crosses it faster than moving in x. SHAP, LIME and Anchors all summarise the whole neighbourhood, and they pick up that slope: y. The counterfactual looks for a single observed point on the other side, and the nearest one happens to lie mostly to the left: x.

Each answer is right about the question it asks. Across all four points the methods agree on the top feature only at point 3: