Curriculum

AI Governance · Model Transparency

Explainable AI (XAI)

XAI techniques reveal how models reach their predictions. SHAP values decompose predictions into per-feature contributions. LIME locally approximates complex models with interpretable surrogates.

The idea

Black-box models are powerful but opaque. Explainability methods answer: "Why did the model predict this?" at global level (which features matter overall) and local level (why this specific prediction).

SHAP (SHapley Additive exPlanations): rooted in cooperative game theory. Distributes the prediction difference from baseline fairly among features. The Shapley value of feature i is its average marginal contribution across all possible feature coalitions. TreeSHAP computes exact Shapley values in O(TLD²) for tree ensembles.

LIME (Local Interpretable Model-agnostic Explanations): perturbs the input, queries the black-box model, and fits a simple linear model to the local neighbourhood. The linear model's coefficients are the explanation. Fast but can be unstable across perturbations.

φᵢ = Σ_{S⊆N\{i}} |S|!(|N|−|S|−1)!/|N|! [f(S∪{i})−f(S)]

Shapley value: feature i's average marginal contribution across all subsets S of features N.

What to hold onto

  • Attention as ExplanationTransformer attention weights are often cited as explanations but are not faithful—different attention patterns can produce identical outputs. SHAP and LIME remain more reliable.
  • Integrated GradientsAttribute prediction to inputs by integrating gradients along a path from a baseline input to the actual input. Axiomatically sound for differentiable models.
  • AnchorsRule-based explanations: "If feature A>0.5 and B='yes', the prediction is stable regardless of other features." High precision but may not cover all inputs.

Live: SHAP Waterfall Feature Attribution

Interactive Shapley value decomposition: base value $E[f(x)]$ pushed by positive and negative feature attributions.

f(x) Prediction: Approved (0.84)

Quick answers

What is Explainable AI (XAI)?

XAI encompasses techniques that make AI model predictions interpretable to humans. Global methods reveal which features the model relies on overall; local methods explain individual predictions. SHAP provides theoretically grounded per-feature attribution using Shapley values from cooperative game theory.

How does LIME explain model predictions?

LIME perturbs the input around the instance to explain, queries the black-box model for predictions on each perturbation, then fits a locally weighted linear model. The linear model's coefficients serve as the explanation, approximating the complex model's behaviour in the local neighbourhood of the input.