What is explainable AI (XAI)?
Explainable AI (XAI) is the collection of methods that make a model’s behavior understandable to the people who build it, use it, audit it, or are affected by it. An explanation answers a specific question about a specific output: which inputs pushed this loan score down, what would have had to be different for the claim to be approved, or which part of the image made the inspection model reject the part.
Explanations come in two scopes. A global explanation describes how the model behaves in general, for example that payment history is its most influential feature. A local explanation describes one decision for one case. Most regulatory and customer-facing needs call for local explanations, while model developers and risk teams rely on both. Explainability is distinct from interpretability, which describes models that are simple enough to be understood by inspection. XAI methods exist because many useful models, including gradient-boosted trees, deep neural networks, and large language models, are black box models that cannot be read that way.
Common explainable AI techniques
| Technique | What it produces | Best suited to |
|---|---|---|
| Shapley values (SHAP) | A contribution score for every feature in a single prediction, grounded in game theory | Tabular models in credit, insurance, and maintenance scoring |
| LIME | A simple local model fitted around one prediction to show which features mattered nearby | Quick checks on text and tabular models |
| Counterfactual explanations | The smallest change to the inputs that would flip the outcome | Adverse action notices and customer-facing decisions |
| Saliency maps | A heat map of the image regions that influenced a vision model | Visual inspection and medical imaging review |
| Surrogate models | A decision tree or rule set trained to mimic the complex model | Global understanding for risk committees |
Why explainable AI matters
Three groups need explanations, and they need different ones. Developers use them to find bugs, such as a model that learned to read a scanner’s timestamp instead of the defect. Risk, compliance, and audit staff use them to show that a model relies on acceptable factors, which model risk management in banking has required for years. Affected people have rights to them: the GDPR grants meaningful information about the logic of automated decisions, US fair lending rules require specific reasons for a credit denial, and the EU AI Act requires that high-risk systems be transparent enough for deployers to interpret their output.
Explanations also change adoption. A maintenance planner who sees that a pump was flagged because of rising vibration and a recent oil analysis will act on the alert. The same planner shown a bare probability will often ignore it.
Explainable AI in practice
A credit union deploys a gradient-boosted model to score auto loan applications. For every decision the system stores SHAP contributions for the top features. When an application is declined, the adverse action letter lists the top three factors in plain language, generated from those contributions. During a quarterly review, the risk team notices that “number of recent inquiries” has become the dominant factor for one branch, traces it to a dealer who runs multiple applications per customer, and corrects the data feed. The explanations served the member, the regulator, and the model’s own maintenance.
What to watch for
- An explanation describes the model, which may still be wrong. Faithful explanations of a bad model make the badness visible, which is their value.
- Post hoc methods approximate. Two techniques can rank features differently for the same prediction, so pick one, document why, and apply it consistently.
- Explanations for large language models are weaker than for tabular models. Showing the retrieved source documents is often more useful than any attribution method.
- Decide who the explanation is for before choosing a method. A compliance officer and a customer need different output from the same decision.
- Explanations can leak. Detailed counterfactuals tell a fraudster exactly what to change, so limit what customer-facing explanations reveal.